Authors

Suggested citation: Batty, R., Wong, A., Florescu, A., and Sharples, M. (2019). Driving EdTech Futures: Testbed models for better evidence. London: Nesta.

Science Practice

Science Practice is a research and design consultancy. Our Good Problems team works with funders to help them identify and prioritise important problems and design effective innovation programmes. Team members on this project were Richard Batty, Andrea Wong, and Ana Florescu.

Mike Sharples

Mike Sharples is Emeritus Professor of Educational Technology in the Institute of Educational Technology at The Open University, UK and Honorary Visiting Professor at the Centre for Innovation in Higher Education, Anglia Ruskin University. He founded the influential Innovating Pedagogy report series and is author of Practical Pedagogy: 20 New Ways to Teach and Learn.

About Nesta

Nesta is an innovation foundation. For us, innovation means turning bold ideas into reality and changing lives for the better.

We use our expertise, skills and funding in areas where there are big challenges facing society.

Nesta is based in the UK and supported by a financial endowment. We work with partners around the globe to bring bold ideas to life to change the world for good.

www.nesta.org.uk

If you'd like this publication in an alternative format such as Braille, large print or audio, please contact us at: [email protected]

Design: Green Doe Graphic Design Ltd

Acknowledgements

This report is the result of generous contributions from many people and organisations. We would like to thank the following for their valuable feedback, insights and time – whether through interviews, workshops, demonstrations or other activities - that have informed this work:

Workshop and interview participants

Kriss Baird, UCL EDUCATE Tom Beresford, Innovation Unit Ollie Bray, LEGO Foundation Alison Clark-Wilson, UCL EDUCATE Sophia Costa, Department for Education Michael Forshaw, Edtech Impact Jen Halmshaw, Department for Education Cathy Lewin, Manchester Metropolitan University Tobias Ley, EDUlabs Mark Martin, South Bank Engineering UTC Raigo Megerild, City of Helsinki Christina Preston, MirandaNet Vanessa Pittard, Mathematics in Education and Industry Anna Maria Rantapero-Laine, City of Helsinki Jeremy Roschelle, Digital Promise Jon Smith, Pobble Catherine Speight, pi-top Eleanor Stringer, Educational Endowment Foundation Jennifer Taylor, Grasmere Primary School, Hackney Lauren Thorpe, Ark Cecilia Waismann, MindCET

Nesta contributors

Toby Baker Celia Hannon Laurie Smith Nancy Wilkinson Joysy John

Executive summary

Educational technology (EdTech) has the potential to transform education but too often it fails to live up to expectations. Cycles of hype and disappointment are common features of the field, and teachers and schools find it difficult to know which EdTech to buy and how best to use it. Conversely, EdTech suppliers often struggle to gain access to schools to test and refine their products.

Evidence is a key part of this problem. Reliable, relevant evidence can enable good decision- making around EdTech. But evidence is often missing, irrelevant, or hard to understand. Although there are several organisations working on this problem in the UK, there is still much to be done.

We have identified five barriers (page 10) to having more evidence-based EdTech.

• The complex context of EdTech infrastructure, services, policy, and pedagogy.
• The wide variety of evidence needed for differing purposes and stakeholders.
• Constraints in schools because of lack of money, time, and skills.
• Inadequate and fragmented advice for schools and teachers.
• EdTech suppliers not generating good enough evidence of what works in practice.

EdTech testbeds are a way to improve this situation by generating and sharing evidence of what works in practice for those who need it. Testbeds have attracted interest in the policy world lately as a way to explore the feasibility of technologies from autonomous vehicles to new financial products. Nesta has recently announced a partnership with the Department for Education to launch an EdTech testbed in England. They will work with partners to build the evidence for what works by connecting schools and industry to test high-potential products.

An EdTech testbed is defined as:

An environment to test and experiment with EdTech in a real-world setting.

This report explores how testbeds can be designed so that the most appropriate EdTech is created and schools are able to use it in the most effective way. We aim for this report to help provide a path to setting up one or more EdTech testbeds in the UK. With the right skills and resources, this would be a highly ambitious yet achievable goal.

We have developed four testbed models that could help with this. These models are intended to act as archetypes, rather than as a detailed blueprint for implementation. A testbed design could also be created by combining two or more of these models. The models are:

• Co-design: EdTech suppliers, researchers, teachers, and students work together to identify educational needs and opportunities, and to develop a combination of technology and pedagogy to address these. (page 20).
• Test and learn: EdTech suppliers work with schools or colleges to rapidly test their product in a real-world setting so they can improve it. (page 21).
• Evidence hub: Educators and policymakers work with EdTech developers and researchers to generate evidence about impact, synthesise it, and disseminate evidence-based advice to guide adoption and scaling. (page 22).
• EdTech network: A network of schools, researchers, and suppliers that share their experience and insights. As well as networking, this may involve training and professional development. (page 23).

These models tackle different parts of the problem. For example, if you want to generate qualitative evidence to inform product concepts or prototypes, a Co-design model would be most useful. But if you want to test the effectiveness of more mature products, then the Evidence hub would be more appropriate. They could also be combined – for example, an Evidence hub could connect with an EdTech network to disseminate its findings and get feedback on them. Together, these models make up different parts of a robust EdTech evidence ecosystem that supports all aspects of EdTech development and use.

When it comes to the implementation of the models, we provide several considerations to aid design and decision-making: (page 26):

• Testbeds should act as a public good that everyone can draw on by sharing findings and data, as well as being transparent about methods.
• Don't just test the technology in isolation - test the pedagogy and implementation as well.
• Have a clear purpose for evidence-gathering and the right methods for this purpose.
• Define the audience for any evidence generated and communicate it to them in a way that they will find useful.
• Make sure that participants in the testbed such as teachers and suppliers have the ability and motivation to take part, and that you have a way to manage friction between different participants.

This report is intended to be useful to anyone considering setting up an EdTech testbed and, more broadly, for anyone involved in EdTech policy. This includes policymakers, researchers, funders, schools and industry organisations.

Research methods

The research for this report was based on:

A literature review

We reviewed relevant academic and policy literature on EdTech evidence. The literature we used is referenced in endnotes and there is a list of recommended further reading in Appendix 2 on page 39.

Interviews with experts

We interviewed nine experts from schools, EdTech suppliers, and academia. Interviewees are listed in the acknowledgements section. We also engaged with a variety of suppliers and other EdTech organisations at the EdTech trade show, BETT.

A workshop with experts

We held a workshop with people from schools, EdTech suppliers, academia, and government on 19 March 2019 to test our understanding of the problem and the testbed models we developed. Participants are listed in the Acknowledgements section.

Case studies of EdTech testbeds

We explored in greater detail five case studies to understand how existing EdTech testbeds work. These were based on desk research and interviews.

1. The challenge: make more effective use of technology in education

There is a wide variety of EdTech on offer to help students, teachers, administrators, and school leaders. Prominent EdTech successes such as the Scratch educational programming environment suggest that it can be beneficial. Systematic evidence also gives hope an evidence review from the Educational Endowment Foundation (EEF) suggests that EdTech overall can offer moderate impact for moderate cost.¹ Technology has the potential to help tackle many challenges in our education system.

But EdTech products have often not lived up to expectations. One OECD report concluded that:

When they [ICTs] are used in the classroom, their impact on student performance is mixed, at best. In fact, PISA results show no appreciable improvements in student achievement in reading, mathematics or science in the countries that had invested heavily in ICT for education.²

The EdTech sector has also gone through many cycles of hype and disappointment stretching back through the decades, including technologies such as educational television and virtual worlds for learning.³

Teachers and schools face difficulties with EdTech. This became clear when speaking with teachers and those involved in EdTech procurement decisions in schools. Lauren Thorpe, the Head of Data and Systems Strategy at Ark Schools, suggested that there are so many EdTech products available that it is difficult to evaluate and choose between the options. She also said it can be difficult to evaluate the credibility and longevity of suppliers, risking leaving schools with unsupported or underdeveloped products. Mark Martin, a computer science teacher and EdTech expert, expressed frustration at EdTech suppliers who promise a lot, but whose products often turn out to have little benefit.

While technology alone will not transform education, an appropriate combination of technology and pedagogy, matched to the context of learning, can have a substantial positive impact. A study of the use of interactive whiteboards in UK schools concluded:

In the hands of a teacher who is interested in developing the independent, creative, thinking skills of their students, [the interactive whiteboard] will be used to further these purposes. . . It's not what you use it's how you use it.⁴

The value of evidence

A significant cause of these problems are challenges in gathering and using evidence. As one report says:

These difficulties [with evidence] can lead to incongruity between technology investments, learning sciences research, and evidence-based decision-making. The disconnections result in large pendulum swings. For example, as school leaders see the appeal of learning technologies, large educational technology investments are sometimes made, usually focusing on hardware and not always capitalizing on what we know about learning. Later, when evidence of improved student outcomes fails to materialize, those investments are characterized as failures. Not enough is learned in the oscillation between high expectations and disappointment as the pendulum swings between ICT optimism and results-oriented reviews of evidence of ICT impact.⁵

This leads us to our core question:

How can you design a testbed so that the most appropriate EdTech is created and schools are able to use it in the most effective way?

Answers to this will have to be pragmatic – working within time, money, and system constraints and balancing tradeoffs between competing objectives.

The context we are focused on is English primary and secondary education. But many of our findings and recommendations will be applicable elsewhere.

What do we mean by evidence?

By evidence, we don't only mean academic studies. We are taking a broader approach where evidence refers to any knowledge based on analysis, observation, or experience that is as reliable and relevant as possible to the practical problems of education.⁶

We're using this broad definition of evidence because different types of evidence can be used for different purposes. Different types of evidence also cost different amounts and require different skills to generate. For instance, a startup trying to test a prototype might carry out usability research because it's cheap and will uncover important usability issues, while a school network trying to test the educational impact of a product might do a randomised control trial across multiple schools.

One approach to using the most appropriate type of evidence for the job is Nesta's Standards of Evidence framework. Although this is useful in understanding different levels of confidence that you can have in different types of evidence, it doesn't take account of using different types of evidence for different purposes. An evidence framework developed for EdTech by the US Department of Education gives an alternative approach. It outlines different evidence-gathering approaches, what they are useful for and what questions they can answer.

The barriers to evidence

If evidence is so important for tackling the problems people in education have with EdTech, why do we see so many problems resulting from inadequate evidence? There are several barriers preventing solutions.

The complex context of EdTech

Although reviews of the evidence suggest that EdTech can be helpful, we can't simply conclude that 'EdTech works'. There is a lot of variation in impact between different studies,⁷ and it's common to find disparate results for a given product in different settings. There are many factors in the environment in which EdTech is used that can affect its effectiveness, such as teaching practices or the amount of time spent on certain tasks. In addition, the speed with which EdTech can change makes it difficult to gather evidence.⁸⁹ So it's not only necessary to answer 'what works' but also consider under what circumstances it works, and for whom.¹⁰ As one report suggests

In order for educators to be able to use technology to deliver education improvement, we need to move away from the seemingly simple question, "What works?” and towards a more systemic approach of learning technology implementation and evaluation.

Furthermore, the evidence needed isn't just about the technology itself. There has been a tendency in EdTech to focus on the technology,¹¹¹² but the report suggests that:¹³

It's rarely possible to disentangle the impact of the learning technology from the effectiveness of the overall instructional system in which it is embedded.¹⁴

This instructional system will include the teaching methods of the teacher and the school they operate in. The impact of any learning technology will also depend on Technological Pedagogical Content Knowledge - the interplay of the teacher's knowledge of pedagogy, their subject, and the technology.¹⁵

The wide variety of evidence that is needed

EdTech evidence is needed for different audiences, about different types of EdTech, at different stages of technological development, in different contexts, for different purposes, and to be communicated in different ways. For example, a startup trying to test a prototype will need different evidence from a multi-academy trust testing the effectiveness of a product across their group of schools. This variety of types of evidence means that a variety of solutions is needed to generate and disseminate it.

Although this variety creates a challenge for anyone wanting to improve the use of evidence in EdTech, there is also an opportunity. By realising that a variety of types of evidence is needed, we can explore how to generate these. In particular we can explore what evidence can be generated that's more rigorous than user reviews, but easier, cheaper, and quicker than randomised controlled trials. See Appendix 2 for a list of frameworks we've come across for different types of evidence.

One report promoting the use of a variety of types of evidence suggests 11 different approaches to evidence-gathering,¹⁶ including:

• Rapid prototyping, which can help with product design.
• Ratings and reviews, which gives the users' perspective on a product.
• Effectiveness studies, which assess the impact of a product or intervention.

In addition to many different types of evidence, there are many different people who can generate evidence. Most actors in the educational system have both a need for evidence and can help with generating evidence.¹⁷ Some examples of this are in the table below:

Constraints in schools

Resource limitations constrain schools' and teachers' ability to use EdTech effectively. According to a 2018 survey of school governors, 71 per cent said funding was the biggest issue facing the school they govern.¹⁸ Many schools also have inadequate IT infrastructure with 33 per cent of secondaries and 60 per cent of primaries saying they are 'well-equipped' with ICT infrastructure.¹⁹

Without adequate funding, it is difficult for schools to utilise EdTech effectively. Even if they have adequate IT, schools need to think about systems issues such as data interoperability and infrastructure. Leaving these issues unresolved can hamper other EdTech use and lead to inefficient processes around the software. As Lauren Thorpe, the Head of Data and Systems Strategy at Ark Schools said: "If the plumbing is broken, it doesn't help to start changing the taps."

If good IT systems aren't in place, it makes it more difficult to gather and analyse data to understand the use and effectiveness of products. Furthermore, if technology has been demonstrated and tested in well-resourced settings it may not work well in a more constrained setting.

Many teachers also struggle with high workload.²⁰ According to a consultation by MirandaNet,²¹ a community of ICT professionals in education:

Another challenge is the pressure on teachers' time that is greater than ever. It is understandable that some teachers just cannot cope with learning how to use digital technologies effectively on top of all their other commitments.

As well as resource limitations, skill limitations are important. According to Michael Forshaw, the founder of EdTech Impact, schools may not know what questions are most important to ask of suppliers or how to run a simple in-school trial to test whether EdTech has the desired impact in their school environment. Teachers' high workload can prevent them from having the time for EdTech-related training.²² Workshop participants mentioned that when there is training, it is too often focused on training for a particular product, rather than the more complex task of learning how to connect the technology with pedagogy.

Partly because of these constraints, schools often don't take a strategic or evidence-based approach to EdTech. According to a British Educational Suppliers Association report,²³ recommendations from other teachers are the most influential source when assessing the efficacy of EdTech products. This is useful evidence, but it would be good if schools were able to use other types of evidence as well.

Workshop participants noted that schools are often under pressure to look for a quick fix and therefore choose EdTech products without a systematic and evidence-based process. These problems not only apply to buying new EdTech, but also to existing EdTech. Schools are often using many different EdTech products and some of them may need evaluating to see if they are worth keeping.

Mark Martin, a computer science teacher and EdTech expert, explained that a lack of a strategic approach can lead teachers to feel that school leadership imposes EdTech on them without having a clear benefit. It can then be challenging for teachers to work out how best to use these technologies.

Inadequate and fragmented advice for schools and teachers

Advice for schools and teachers on EdTech is fragmented, with a lack of clear sources of guidance. There are some sources of advice, such as the Educational Endowment Foundation's guide on Using Digital Technology to Improve Learning. There are also websites that review EdTech products based on structured teacher reports such as the Educational App Store and EdTech Impact. But more is needed.²⁴ One teacher quoted in the MirandaNet consultation summed up this problem:

Teachers need time to adequately integrate a new technology into their practice. There is a lack of national support that is exacerbated by the vastness of the internet, not knowing where to find trustworthy support and generic ICT CPD [Continuing Professional Development]. Teachers want to understand how a piece of technology can be used in their subject or specialism. Generic, de-contextualised CPD doesn't appeal.²⁵

Participants in our workshop suggested that the education sector, in general, is fragmented, with a lack of guidance at the national level and strategy at the school level. They also mentioned that despite the good work of existing organisations focused on evidence, these organisations did not coordinate with each other well.

EdTech suppliers not generating good enough evidence

EdTech suppliers often don't generate the evidence that schools need to make good decisions. The EDUCATE programme at UCL is working on solving this problem by providing training and advice on evidence for EdTech suppliers. They have described some of the challenges suppliers face in a paper about their programme, which we have drawn on for this section.

Partially, there is a challenge of suppliers not being motivated to do research. There is a perception that doing research will be too slow for the fast cycles of product development.²⁶ Furthermore, because EdTech products can go straight to market without evaluation, this leads to some suppliers believing that evidence is unnecessary.²⁷

On the other hand, suppliers do have an incentive to generate evidence. This is especially true of small companies which have a small marketing budget. They may work on generating evidence so that they can give schools confidence in their products.

Even suppliers committed to evidence can find it difficult. Although large companies often have in-house research teams, smaller ones often don't, and they find it hard to engage with the research community or work out how best to generate their own evidence.²⁸ There can also be barriers, in particular for small companies being able to test their products. Large companies such as Google or Apple may have 'certified educators' experimenting with their products in classrooms. But it can be harder for startups.²⁹ Furthermore, the EDUCATE programme found it difficult to get academics engaged in helping EdTech suppliers, because of their workload and perhaps their incentives.³⁰

Current and past initiatives to address this challenge

There are several initiatives that have worked or are currently working in the UK on evidence in education. Some are focused on EdTech specifically, while others focus on education evidence in general. These include:

Testbeds as a possible solution

EdTech testbeds are a potential solution to the problems outlined above. Testbeds have been an area of interest in the policy world recently, and are an approach that Nesta has been exploring both in EdTech and in other domains.³⁶ In partnership with Nesta, the Education Secretary recently announced a fund that "will be supported by a group of schools and colleges selected to aid the development, piloting and evaluation of innovative technology."³⁷ Nesta and the Department for Education will be setting up a testbed of schools in England to help develop technology products and share what works with schools. Testbeds have also been set up in other areas of policy, such as the NHS Test Beds programme for digital technologies.

We have kept our definition of EdTech testbeds broad so that we can understand all the options:

EdTech testbed: An environment to test and experiment with EdTech in a real-world setting.

This definition has the following parts:

• An environment: This could be a physical environment such as an individual classroom, a school, or a group of schools. Or it could be an organisational or virtual environment.
• To test and experiment: a testbed can be both for testing whether and how EdTech works as well as experimenting with it and improving it.
• With EdTech: a testbed should test not only an EdTech product but all the factors that are needed to make it a success such as teaching materials, teaching techniques, and how it can be integrated into existing systems.
• In a real-world setting: a testbed should not be an artificial environment such as a lab, it should involve real schools.

As highlighted earlier in the report, there are several initiatives in England that are currently working on evidence in education. However, most of these don't meet the requirements to qualify as an EdTech testbed. For example, UCL EDUCATE doesn't explicitly engage with schools, while the Research Schools Network (a collaboration between the Educational Endowment Foundation and the Institute for Effective Education) is focused on education more generally rather than EdTech.

EdTech Impact and Educational App Store are testbeds in that they provide a way for teachers to test and give feedback on educational products. But, with the right skills and resources, there is scope for other types of EdTech testbeds to emerge in the UK.

2. Learning from case studies of existing testbeds

To inform the development of the EdTech testbed models presented in this report, we conducted rapid research to identify examples of existing EdTech testbeds from around the world. We found 30 diverse examples that you can see listed in Appendix 3: List of testbed examples.

We chose five testbeds to present as case studies. These are iZone (New York City, USA), MindCET (Israel), Digital Promise (USA), European Schoolnet (Europe), and EDUlabs (Tallinn, Estonia). These were selected because they are actively trying to address some of the EdTech evidence challenges outlined in this report, and they offer a diversity of approaches across a range of countries.

In the following section, we summarise these case studies, along with the key insights we got from researching them. You can read the full case studies in Appendix 1: Full testbed case studies.

iZone

New York City, USA izonenyc.org

Description

New York City's Innovation Zone (iZone) is an education reform programme of the city's Department of Education that supports technology integration and innovation for personalisation. Launched in 2009, the initiative was intended to catalyse and scale innovation by providing a structure for designing and testing new learning models, connecting technology providers with educators, and working with policymakers to remove barriers and build capacity for innovation. The programme serves the largest school district in the USA, which has over one million students.³⁹ Nearly 300 primary- and secondary-level schools in the district participated in the initiative,⁴⁰ which includes a grant programme to support innovative projects using learning technologies to address challenges across multiple schools, a blended learning programme with a dedicated teacher training component, a personalised learning programme, and a programme called Innovate NYC Schools to foster EdTech innovation by connecting schools and EdTech suppliers.

Purpose

Since 2014, under the Innovate NYC Schools programme, iZone has partnered with the Learning Assembly on the Short-Cycle Evaluation Challenge (SCEC). Supported by the Bill and Melinda Gates Foundation, the Learning Assembly is a network of seven not-for-profit organisations testing EdTech, collaborating with educators to identify which EdTech works, providing EdTech suppliers with feedback, and sharing evidence of learning outcomes to improve what is offered on the EdTech market.

The SCEC aims to evaluate 'who EdTech products work for, when, and under what circumstances'⁴¹ by matching teachers with EdTech suppliers to pilot products with potential to address student needs. In running the SCEC, iZone concentrates on 'problems of practice'⁴² to understand school needs, engages all EdTech stakeholders and especially strengthens teacher input to improve the quality of current and future EdTech products, and builds school capacity to make better, more strategic decisions about their needs in relation to EdTech. Although the programme benefits EdTech suppliers by providing an opportunity for them to test their products, iZone's primary vision for the SCEC is to provide educators with the information they need to make decisions about using EdTech, when they need it.

The development of the SCEC builds on a 2013 programme called the Gap App Challenge, which invited 13 suppliers to design desktop and mobile apps for enhancing mathematics teaching, learning, and engagement in middle schools, and to partner with an iZone school to pilot them. The programme was intended to provide feedback loops between educators and EdTech suppliers and to overcome barriers to procuring EdTech through requests for proposals.⁴³ However, suppliers with earlier-stage product ideas were less able to contribute or gain value from the programme.⁴⁴ In response to this learning, iZone developed a resource for EdTech suppliers that divides EdTech product development into three phases: problem and product validation, piloting, and scaling support to more schools.⁴⁵ They designed the SCEC to engage with EdTech suppliers who have passed through the problem and product validation phase and are ready to pilot their product.

Testbed design

Through the SCEC, iZone is developing a 'rapid-cycle' EdTech piloting process that is less resource-intensive than randomised control trials yet more robust than product reviews. For 12 weeks of the school year, teacher teams of four to six educators test-drive a technology product - 'an app, software or web-based program'⁴⁶ – that will help them meet a shared need. Research is conducted by iZone's research partner. These have included Edtechnos Evaluator, an EdTech advisory firm comprised of consultants from Good Harbor Partners and evaluators from John Hopkins University, and the Center for Children and Technology.

For schools to participate, teams of four to six teachers of any mix of grade levels or content areas within that school identify a common need within their teaching practices. By applying to the SCEC, they commit to piloting an EdTech product that addresses this need. iZone reviews submitted SCEC applications and conducts in-school interviews with selected school candidates to gain an in-depth understanding of their expressed needs and contexts. Schools are required to provide student access to desktop computers, laptops, or tablets for the length of the pilot, and this access must enable all students in a class to be online at the same time.

Any EdTech supplier ready to pilot a product can apply to test their work through the SCEC. If selected for participation, EdTech suppliers are required to provide free access to their product for the entire school year, teacher and researcher training and support, a dedicated representative, feedback required by the research process, and product usage data.⁴⁷

iZone evaluates EdTech suppliers applying to the SCEC with a rubric⁴⁸ containing criteria⁴⁹ related to teacher needs, product readiness, and supplier capacity. This enables them to identify a shortlist of EdTech suppliers with the greatest potential to meet each school's needs, context, and available hardware. SCEC teacher teams have an opportunity to try out⁵⁰ options on their shortlist ahead of deciding which EdTech to pilot. Finally, schools and EdTech suppliers participate in a virtual matching session including product demos followed by 20-minute question periods. Schools then rank their choices and iZone assigns school-supplier matches.

Implementation

During pilots, EdTech suppliers have the opportunity to collect data through classroom observation⁵¹ and to improve their products by collaborating with teachers during structured feedback sessions.⁵² From the research partner, they also receive an independent evaluation of their product's efficacy based on the pilot.⁵³

As one of the previous research partners describes,⁵⁴ the evaluation research approach is supposed to result in findings that apply to specific rather than generalised implementation of an EdTech product. For example, Edtechnos Evaluator used a set of measures that could be flexibly combined to test a variety of different EdTech products. This kit-of-parts approach allowed them to adapt to different data availabilities and needs. Depending on the pilot, researchers could collect data on:

• Student learning, through quantitative and qualitative data on grades and teacher-made tests, programme assessments like embedded quizzes, teacher perceptions, and standardised assessments.
• Student and teacher engagement, through classroom observations, an online teacher survey with rating exercises and open-ended questions, teacher focus groups and interviews, bi-weekly reflections, online student surveys with a mix of rating exercises and open-ended questions, and student focus groups and interviews.
• Student and teacher satisfaction, through similar methods as for engagement.

These researchers summarised their findings in two formats: a one-page short report and a longer, more detailed implementation report.

Impact

Weaknesses. The iZone has shown itself to be somewhat vulnerable to changes in politics and leadership - unfortunately, the initiative suffered a 93 per cent budget decrease from 2013 to 2017.⁵⁵ This may illustrate the importance of establishing funding models for EdTech testbeds that are not tied to any one political figure or party if such programmes are to persist through change. However, the SCEC programme may be somewhat protected by its external funding.⁵⁶

Key takeaways

Strengths: The SCEC's design is teacher-centred and allows for both design research by EdTech suppliers, and external evaluation by dedicated researchers. Sets of data collection techniques used by the researchers allow for flexibility in designing pilot-specific studies that test specific, rather than generalised, implementation of a single software product.

Weaknesses: Although the SCEC programme appears to be successful, the iZone itself has been vulnerable to changes in politics and leadership. Unfortunately, iZone reportedly suffered a 93 per cent budget decrease from 2013 to 2017.⁵⁷ However, because SCEC is externally funded by the Gates Foundation,⁵⁸ the programme may be somewhat protected.

Related resources

The iZone website

Centre for Public Impact (8 April 2016). Case Study: New York City Innovation Zone

MindCET

Israel www.mindcet.org/en

Description

MindCET is an EdTech innovation centre operated since 2012 by the Centre for Educational Technology (CET), an Israeli not-for-profit organisation that aims to advance education in Israel and beyond by working to ensure EdTech serves learning and teaching needs. MindCET brings together entrepreneurs, educators and researchers through incubator and accelerator programmes, training and network-building activities, research and publishing, and investment in early-stage EdTech ventures. It is supported by government, university, and industry partners. Since 2016, MindCET has collaborated with EdTech UK and UK Israel Tech Hub on the UK Israel EdTech Task Force.⁵⁹ Overall, MindCET aims to build partnerships to create a 'new educational paradigm' that properly benefits from advances in technology, to mobilise Israel's startup and innovation culture toward solving problems in the education field, and to initiate learning and improvement within the CET.

These offerings allow MindCET to work across different levels of education and to support the needs of educators, learners, and EdTech creators by enabling a feedback and development cycle centred around improving the creation of EdTech products and the way they are integrated into education systems.

Purpose

MindCET has three main programme areas: an accelerator for startups looking to enter the EdTech industry, a teacher-oriented programme with opportunities to receive training and provide feedback on EdTech products in development, and a challenge-driven incubator residency supported by industry partners.

MindCET facilitates access to appropriate field trial and pilot opportunities for participants in the incubator and accelerator programmes. They also invite EdTech startups and teachers to partner and learn from one another through the Teacher Early Adopters of MindCET (Τ.Ε.Α.Μ.) network of early-adopter teachers who are trained and accredited through the MindCET's EdTech Startups and Teachers Alliance (E.S.T.Alliance). MindCET Fellows positions teachers themselves as entrepreneurs and provides an opportunity for them to develop an EdTech tool that addresses pedagogical and educational challenges they have identified from within their teaching practice.

Testbed design

The T.E.A.M. programme illustrates MindCET's position as a third-party facilitator within the EdTech ecosystem. The programme involves training teachers as early adopters of EdTech products by empowering them to test in their own classrooms and partnering these teachers with EdTech suppliers with pre-market working prototypes (i.e. early-stage, post-MVP, having tested with users) who are looking for opportunities to test in real-world settings. Teachers gain an understanding of the context for testing, and an opportunity to become involved in the EdTech development process. EdTech suppliers gain insights into how their proposed solution performs in practice.

The T.E.A.M. programme takes place over four to six months and includes both online and in-person events. Teachers who join the programme are assigned to groups with a representative from the MindCET team. They receive some initial training through online sessions, where they learn about what is involved in testing emerging EdTech products. This includes exposing them to different examples of EdTech and teaching them how to select the solution they will be testing based on their own teaching needs. At an in-person showcase event, startups screened by MindCET and invited to test through the T.E.A.M. programme pitch to teachers. This event also includes roundtable discussions for teachers and suppliers to learn about product specifics in greater depth. Teachers shortlist products that meet their needs, and through consultation with MindCET representatives, eventually choose one work with. Then MindCET facilitates longer-term partnerships between teachers and EdTech suppliers for testing.

Implementation

Testing and piloting are carried out according to a methodology and a set of tools developed specifically for the education ecosystem. These are designed to consider pedagogy, technology, and implementation. MindCET's focus is product-oriented, with the emphasis on trying out a new product, providing feedback, and understanding the pedagogical added value. Having completed training through the Τ.Ε.Α.Μ. onboarding process, teachers themselves implement testing in their own classrooms over a period of four to five months. During this time, they continue to share experiences with MindCET, other Τ.Ε.Α.Μ. teachers testing the same EdTech product, and the supplier developing the EdTech being tested - who in some instances also observe classroom use of their product. Teachers also provide feedback and data to MindCET researchers through a set of instruments designed to collect both quantitative and qualitative information. MindCET researchers process this information and prepare reports for both the teachers and EdTech suppliers involved in a test.

Impact

Through the Τ.Ε.Α.Μ. programme, MindCET enables both EdTech startups and teachers. EdTech developers learn how to test their product and gain insights from trying out their products with practicing teachers in real classroom settings. Teachers learn how to select, test, and contribute to improving EdTech products. Their involvement in the EdTech development process helps dispel fears about integrating emerging technology in their teaching, and since 2016 MindCET has noticed a change in culture in teachers:

"It was really hard for [teachers] to understand how to test a product that was not fully ready. Today... they find it extremely enriching, the fact that their feedback will, in a way, shape the way the project is going to end up."

– Cecilia Waismann, Academic Director, MindCET

Key takeaways

Strengths: MindCET enables two-way exchanges between early-stage startups and teachers, building relationships between these stakeholder groups and empowering both to carry out testing for the sake of improving the quality of EdTech tools available on the market.

Weaknesses: Τ.Ε.Α.Μ.'s data collection instruments, though refined through programme iterations, have yet to be formalised. The focus of the programme makes it potentially less suited to providing policy recommendations, although other R&D offerings within MindCET address this gap.

Related resources

The MindCET website

Ramiel, H. (2017) User or student: constructing the subject in Edtech incubator. 'Discourse: Studies in the Cultural Politics of Education, 1-13.'

Digital Promise

USA digitalpromise.org

Description

Authorised by the US Congress in 2008⁶⁰ as the National Center for Research in Advanced Information and Digital Technologies and launched in 2011,⁶¹ Digital Promise is an independent, bipartisan not-for-profit organisation. Digital Promise was founded to 'realise the potential of learning technology' by engaging with educators, researchers, technology firms, and entrepreneurs to: identify breakthrough technologies, learn faster what's working and what's not,⁶² and transform the market for learning technologies. Today, its mission is to 'accelerate innovation in education to improve opportunities to learn',⁶³ with a focus on addressing Americans' lack of equity in access to, participation in, and use of technology for learning at all ages, in school and beyond. The organisation receives operating funding from a set of core funders and also pursues partnerships and grants to support individual projects.

Purpose

In pursuit of its mission, Digital Promise engages in activities related to network building among educational leaders and innovative school districts; documenting and disseminating innovations to build capacity for change; facilitating access to existing research tools, data, and studies; and expanding opportunities to engage students in active learning through problem-solving and making. More recently, in 2017 Digital Promise hired leading researchers to develop a responsive research programme serving the needs of educators and students, EdTech providers, and policymakers.⁶⁴

Digital Promise's League of Innovative Schools often acts as a testbed to try out innovative approaches that improve education. The League convenes over 100 school districts⁶⁵ across the US. Together, they identify challenges that they share, and seek and share ways to address them. Accordingly, members often partner with entrepreneurs, researchers, and relevant experts to pursue research that supports decision-making. These activities may involve EdTech but are primarily focused on finding solutions to a context-specific problem.

Testbed design

With the League of Innovative Schools, testbed activity comes in the form of collaborative, consultative engagement at the school district level - with school districts often banding together to address the same challenge. Digital Promise aims to understand what decisions school districts and their schools are trying to make. It then designs research programmes and trials to support them in making informed judgements and taking appropriate actions. This may include helping school districts select appropriate metrics and design evidence-gathering programmes to collect these. Digital Promise trains district staff so that they are empowered to continue the work and advance it to further address their own needs. They help build toolkits, frameworks, and helpful resources, and share these across schools in the League to ensure benefits go beyond a single project implementation.

Implementation

One challenge the League of Innovative Schools has addressed through this approach is data interoperability. With so many different digital technologies being used for teaching and learning, schools are looking to better understand the data they could be accessing, aggregating, and analysing to help them make better decisions about how and what they teach. Due to federal-level legislation, educators have been required to adopt a data-driven approach to decision-making and evaluation since 2001; however, making sense of the abundance of data that schools collect and then acting on it to improve student outcomes has proven challenging.⁶⁶

To address this challenge, Digital Promise provides research expertise to help figure out what metrics would support school decision-making, and how schools could go about collecting this data through the EdTech products they are already using. Working with individual school districts, Digital Promise researchers transfer techniques to school district staff so that they can continue to advance the work and improve their ability to monitor and evaluate their own programmes. Digital Promise has summarised lessons learned so far about data interoperability in a report that recommends actions and sets the stage for advocacy work in this area.⁶⁷

Impact

Digital Promise provides strategic and research leadership through their engagements with school districts. Previously, their research activities were mostly translational in nature, taking available research done elsewhere and helping schools make use of it. However, the organisation has recently expanded its capacity to carry out original research. For now, the League of Innovative Schools appears to be effective at surfacing common problems, and cooperating and collaborating to solve these through projects and advocacy.

Key takeaways

Strengths: Digital Promise uses a network of school leaders to enable and build consensus around a prioritised set of shared challenges, and to attract funding for related projects. Its approach is context-sensitive and highly attuned to individual school needs, while ensuring learnings are shared across a wider community of research and practice.

Weaknesses: The model is not explicitly designed for testing EdTech and is not concerned with standardisation. However, it does build capacity among League of Innovative Schools districts for scaling approaches and methodologies.

Related resources

Digital Promise website.

European Schoolnet

Europe www.eun.org

Description

European Schoolnet is a not-for-profit organisation that supports innovation in learning and teaching across a network of 34 national education ministries throughout Europe. It supports a wide range of stakeholders within the education ecosystem by engaging education ministries, schools, the European Commission, IT industry (including EdTech suppliers), and experts and by targeting policymakers, teachers, school leaders, students, and research and development organisations. European Schoolnet is closely involved with the European Commission's programmes in education, research and technology. It pursues multi-stakeholder projects as well as larger bilateral agreements to support effective integration of technology in education with a focus on ICT and digitisation. This work involves identifying and testing promising innovations, sharing evidence of impact, and helping to mainstream advancements in teaching and learning for an inclusive education system. European Schoolnet supports inter-school collaboration, disseminates good practice, investigates new education models of technology-enhanced learning, and provides professional development for educators by directly and indirectly developing and supporting a network of schools already taking innovative approaches to teaching and learning.

Purpose

Currently, European Schoolnet's focus areas include digital citizenship, STEM education, professional development, school networking, and evidence for innovation. This last area is concerned with collecting evidence and data regarding innovations in education, with the specific aim of using this to make policy recommendations and to scale. Accordingly, European Schoolnet initiates both translational and evidence-gathering research, and facilitates experimental pilots. Through large-scale 'policy experiments' using randomised control trials, they test new tools and resources in a way that allows them to be taken up more widely. So far, this approach has been used to test MENTEP, a tool for teachers to self-assess their progress in Technology-Enhanced Teaching, and Teach-Up, a set of online course modules for teacher training.

The European Schoolnet's Future Classroom Lab in Brussels complements these activities. Primarily, it is a space for convening education stakeholders and showcasing innovations in education, including current and emerging EdTech. The lab is independently funded and supported through partnerships with the ICT industry. Through the Future Classroom Lab, the European Schoolnet also runs a validation service. Based on findings from the scale-oriented Living School Labs project, the validation service was initiated to help EdTech suppliers and researchers run small- and large-scale pilots in schools in a way that aligns with teacher needs for testing support.

Testbed design

The Future Classroom Lab's validation service was designed to meet the needs of EdTech suppliers and teachers. Companies expressed a need for faster research and a variety of testing models, and a willingness to provide software licenses. Teachers expressed interest in free software and opportunities for professional development, but also warned of constraints on their time given their normal workloads.

In response, the Future Classroom Lab's validation service provides a way for organisations or individuals (usually related to European Commission projects, national education ministries and policymakers, or industry) to propose validation pilots that follow a range of different evaluation methodologies to test either design, implementation or effectiveness of EdTech products in K-12 school settings. These methodologies may be 'heavier' – more costly, longer, larger-scale, and requiring more resources – or 'lighter' - less costly, shorter, smaller-scale, and requiring fewer resources. European Schoolnet provides guidance on validation methodology, operational process, validation scenarios, and tools and templates to would-be testers as well as consultancy services from validation experts to help EdTech suppliers and research organisations design and scope an appropriate validation that considers evidence needs and how to motivate teacher participation. Testers can then apply to access the European Schoolnet's network of schools across Europe to launch their validation at their organisation's or project's own expense.

Implementation

The service provides a detailed manual that helps would-be testers determine their testing goal, plan and design an evaluation, determine an approach, and run an EdTech validation in schools. The manual emphasizes trade-offs between sample size and cost,⁶⁸ and recommends a cooperative research approach wherein research is carried out in cooperation with schools. The Future Classroom Lab recommends this method to those intending to design and run school pilots because it centres around addressing education practitioner-identified problems,⁶⁹ does not require research expertise to understand it, and for these reasons appeals to teachers. Furthermore, it allows for flexible and rapid gathering of evidence that can support both policymakers and EdTech providers.

Impact

The European Schoolnet has run seven large-scale validation pilots since 2010. These were all a result of partnerships with large-scale corporate partners like Samsung, Acer, and the educational gaming supplier Triseum. Some of the validation and evaluation methods, such as surveys, have also been used in other EUN projects.

Key takeaways

Strengths: This programme offers a flexible approach to testing that accommodates different purposes, sample sizes, durations, research methods, and data collection instruments. A detailed guidance and consultancy service facilitates well designed tests that meet the needs of both research initiators, (whether EdTech suppliers or academics), and schools.

Weaknesses: Although the model purports to allow for the running of pilots of varying scales, the limited number of large-scale pilots completed since 2010 seems to indicate that barriers within this model prevent testing with smaller or earlier-stage suppliers on a smaller scale.

Related resources

European Schoolnet website.

Balanskat, A., Wastiau, P., Leontaraki, I., Durando, M. and Ayre, J. (2014, September) 'Validation Manual: How to design and run school pilots.' Brussels: European Schoolnet.

EDUlabs

Tallinn, Estonia ceiter.tlu.ee/living-labs

Description

EDUlabs is an EdTech research, training, and development initiative led by CEITER, the Learning Analytics and Educational Innovation research centre at Tallinn University in Estonia. It aims to support evidence-based educational innovation through an integrated research and teacher training programme. The programme draws on living labs methods, taking an innovation incubator approach that starts with a more controlled setting before scaling to test with a wider target group.

EDUlabs works closely with educators in Estonia to co-create EdTech innovations. It aims to provide a testbed for education and EdTech research to enable scaling beyond single classrooms. It is supported by funding from the European Union's Horizon 2020 research and innovation programme.

Purpose

EDUlabs aims to support evidence-based educational innovation through an integrated research and teacher training programme. The programme draws on living labs methods, taking an innovation incubator approach that starts with a more controlled setting before scaling to test with a wider target group. Research is led by teachers, who participate at all stages in the research process. The programme is designed to test teaching and learning methods in STEM and how they affect student learning.

Testbed design

EDUlabs is for refining an educational innovation or making it work better – not about making a judgement that it does or does not work. Each of these programmes follows a four-stage model.⁷⁰ This begins with researchers and teachers co-designing a research study. Then, researchers test an intervention within the controlled environment of the research centre's classroom lab. Through a teacher training programme that makes use of both in-person training and online engagement, the programme translates and sustains testing in real-world classrooms.

Stage 0: Inventing

University researchers initiate research studies, but design them through a co-creation process with teachers.

Stage 1: Investigating

Researchers invite teachers to bring their class to a controlled environment in the university to try out a novel educational intervention during a 'project day'. Teacher participation at this stage is more passive, as lessons are designed by researchers. This stage allows researchers to experiment with evidence-gathering techniques.

Stage 2: Scaling

Teachers become more actively involved in this stage, joining a six to 12 month in-service training course with an online component and once monthly meetings at Tallinn University. During this engagement, teachers get involved in refining and co-creating lesson plans, teaching materials, tests, and other digital learning resources that were proposed in Stage 1. Then, they try implementing the novel intervention in their own classroom, and report back to the university researchers via feedback tools. These feedback tools include pre- and post-trial standardised tests to investigate aspects like self-regulation, motivation and engagement, and feedback forms to collect qualitative evaluations following the teacher inquiry into student learning model.

Stage 3: Sustaining

Teachers continue to connect with one another over an online community, eDidaktikum, which they were introduced to in Phase 2. In that phase, they already co-created digital learning resources in which learning analytics are embedded. So, in Phase 3, it is possible for researchers to track how learning happens in many classrooms without requiring teacher involvement in the corresponding Phase 2 training, enabling further diffusion. This phase has yet to be piloted but will run for the first time in 2019.

In future, the programme may introduce filters or checkpoints to help refine projects and prioritise which ones advance through each subsequent stage of their model.

Implementation

In the Robomath EDUlab, educators teaching grades three and six are using robots to visualise math concepts and encourage engagement with the subject. Researchers in educational technology co-designed the study with practicing teachers and ran an initial pilot study to inform further development of the Robomath intervention. In the 2018-2019 school year, the intervention was again tested, but this time the pilot included more than 2,000 students in 15 per cent of the schools in Estonia. When the programme reaches Stage 3, Sustaining, over 30 per cent of Estonian schools will be able to access to Robomath.

The Outdoor Learning EDUlab enables learning outside of the classroom, using mobile learning to enable outdoor education. In contrast with Robomath, six schools are involved in this project. Teachers from these schools helped to design mobile outdoor learning scenarios integrating different technologies like sensors, robotic devices and mobile apps to teach using a problem-solving approach across natural sciences subjects. They are now testing these with students within their teaching practices to build an evidence base for the intervention's continued use.

Currently, EDUlabs is pursuing two other programmes in addition to these: Digimath and Smart School House.

Impact

In general, engaging teachers has so far been easy for EDUlabs, with the most recent teacher training programme being fully subscribed. However, travel makes involving teachers from regions beyond Tallinn more difficult. Another challenge has been engaging both teachers and researchers, who each have their own priorities and expectations.

Key takeaways

Strengths: Strong co-creative approach that takes its research lead from teachers and follows a well-developed structure that accommodates development, testing, and scaling of various educational innovations, not just EdTech.

Weaknesses: Does not result in evidence that helps decision-makers judge whether or not an intervention works outright. As the programme is tied to academic timelines (the progress of researchers' PhD projects), it is potentially slower than the timelines by which industry operates.

Related resources

The EDUlabs website.

Ley, T. (19 October 2018) EDULabs: Co-creating Educational Innovations with Estonian Schools.

CEITER. CEITER Educational Living Labs.

Appendix 2: Recommended reading

We found the following reading particularly useful in writing this report providing a good next step for people interested in finding out more.

Means, B., Shear, L. and Roschelle, J. (2015) 'Using technology and evidence to promote cultures of educational innovation: The example of science and mathematics education.' Menlo Park, CA: SRI International.

Means, B., Murphy, R. and Shear, L. (2017) 'Pearson | SRI Series on Building Efficacy in Learning Technologies. Vol. 1 Understand, Implement & Evaluate.' London: Pearson.

Scanlon, E., Sharples, M., Fenton-O'Creevy, M., Fleck, J., Cooban, C,. Ferguson, R., Cross, S. and Waterhouse, P. (2013) ‘Beyond prototypes: Enabling innovation in technology-enhanced learning.' Open University, Milton Keynes.

Cukurova, M., Luckin, R. and Clark-Wilson, A. (2019) Creating the golden triangle of evidence-informed education technology with EDUCATE. ‘British Journal of Educational Technology.' (2019).

Educational Endowment Foundation (2019) 'Using Digital Technology to Improve Learning: Guidance Report.'

There are also several useful frameworks that outline different methods for generating evidence.

Means, B. and Anderson, K. (2013) 'Expanding Evidence Approaches for Learning in a Digital World.' Office of Educational Technology, US Department of Education, p. 78.

Nesta's Standards of Evidence.

Appendix 3: List of testbed examples

Endnotes

58 Victoria Embankment London EC4Y ODS +44 (0)20 7438 2500 [email protected] @nesta_uk www.facebook.com/nesta.uk www.nesta.org.uk

Nesta is a registered charity in England and Wales with company number 7706036 and charity number 1144091. Registered as a charity in Scotland number SCO42833. Registered office: 58 Victoria Embankment, London, EC4Y ODS.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.