Interviews

This section provides details on the method used for this project following the COREQ guidelines for reporting of qualitative research. We carried out nine semi-structured interviews with Utilita customers and conducted a simplified thematic analysis of their answers based on the interview transcripts.

Research team

Interviews were led by one of three Nesta team members with experience of conducting research interviews, usually joined by another Nesta staff member to assist in note taking, and always by a Utilita staff member. Participants were not known personally to the interviewers, although they may have had a brief conversation with the Utilita staff member during recruitment (see below).

Study design

The sampling frame was Utilita customers with smart prepayment meters who were invited to participate in Power Payback. Utilita classified Power Payback participants into high, medium, and low engagement based on the proportion of events they opted in to. Further groups were identified for customers where Utilita had a record of the presence of a low-carbon technology such as solar panels or a heat pump, and for customers who did not opt to take part in Power Payback at all. Customers in all these groups were contacted by telephone to invite them to participate. We aimed to recruit 8-12 participants, spread evenly across levels of engagement. If they agreed, a time slot was booked, and they were asked to complete an online consent form. A £50 incentive voucher was offered as a thank you for taking part in the interview.

At the halfway point of the study, 18 participants had consented to be interviewed and arranged a slot, but only four had joined an interview. To improve the likelihood of ultimately recruiting enough participants, Utilita then contacted larger groups of customers by email asking them to express interest in participating. These customers were called back to arrange an interview. This resulted in a further five interviews being conducted, leading to a total of nine (see table 1). Ongoing analysis (see details below) suggested that data saturation had been reached.

Interviews were conducted online on Microsoft Teams and lasted up to 45 minutes. Prior to each interview, Utilita prepared a briefing sheet with some background information on the participant and their participation in Power Payback, as summarised in table 1. Nesta and Utilita team members joined a brief meeting in advance of each interview to become familiarised with this data. Interviews followed a topic guide, but were semi-structured and so deviated from the guide to explore points raised by participants, where appropriate. Interviews were recorded and automatically transcribed by Microsoft Teams, and notes were taken during the course of interviews to record immediate reflections.

Qualitative data analysis

Thematic analysis of the interviews was conducted using the qualitative software Quirkos. Following a general grounded theory approach to coding, transcripts were imported into the software, and an initial two interviews were blind-coded by three members of the Nesta team. There was good alignment between coders and, following discussion to maximise consistency in approach, the remaining transcripts were divided between the team. Themes emerged from the codes that were developed both deductively from the research questions and inductively from the raw data. We summarised these themes in the report, and used them to help identify illustrative quotations.

Forum topic and sentiment analysis

We also conducted a forum sentiment analysis to provide additional insight into experiences of, and attitudes towards, DFS offerings from a broader range of perspectives – such as credit customers, and households which are more likely to have low-carbon technologies such as domestic batteries or electric vehicles. This analysis was conducted using data extracted from MSE forum posts. Sampling bias resulting from factors such as who chooses to post on MSE forums, and why, means that forum data does not provide a representative view of DFS attitudes and experiences. Observations such as whether certain topics are raised at all can provide helpful context.

The earliest available post is from 2003 (although in practice the earliest post drawn on is from 2022 as that is when the DFS began), and the most recent data is from 3 June 2024. We searched for posts mentioning the following terms: ‘demand flexibility service’, ‘DFS scheme’, ‘Power Payback’, ‘Saving Sessions’. We included Saving Sessions (the DFS offered by Octopus) because it made up the largest share of demand shift in the winter of 2022/23. These terms appeared in 24 posts, which together with their total of 2,166 replies provide the corpus for this analysis. To identify themes of conversation, posts were broken into individual sentences, small sentences removed, and topic modelling (BERTopic) used to cluster sentences by topic. A sentiment analysis model was used to classify sentences by sentiment (positive, negative, neutral). By aggregating the data by theme and sentiment we were then able to draw on it in the relevant sections of this report. The analysis code is available in this GitHub repository.

We are extremely grateful to Utilita for partnering with us to explore their customers' experiences of taking part in Power Payback. In particular, we would like to thank Rebecca Clark, Dimitrios Stefanoglou, and Archie Lasseter for all their work to enable the project. Many thanks also go to the interview participants for taking the time to talk with us.

Nesta would also like to thank Camille Stengel, Head of Qualitative Research and Quality Assurance, for her assistance on this report.

Michael Fell’s work on this project was partially funded by the UKRI Energy Demand Research Centre (EP/Y010078/1).