In a world where attention is a currency, it makes sense to spend it wisely. The first people to spend a lot of time on the internet saw this very clearly. In Howard Rheingold's 1993 book Virtual Communities, one of the earliest works to chronicle the reality of life online, he laid out two rules for the coming age: "Rule Number One is to pay attention. Rule Number Two might be: attention is a limited resource, so pay attention to where you pay attention."

Unfortunately, as a growing body of scientific work is showing, paying attention to where we pay attention is not something we are very good at. You know how it goes: one moment you're reading or driving, the next you're off in a daze, thinking about what you should have for lunch, or running through to-do lists in your head. Because we only notice we have drifted off when we awake with a start some time later, it is easy to write these lapses off as trivial. In fact, this kind of mind-wandering is how we spend a large proportion of our lives.

In 2010 Harvard researchers Matthew Killingsworth and Daniel Gilbert developed an app that pinged people throughout the day to ask whether they were thinking about what they were doing. Responses varied (work provoked the greatest proportion of mind-wandering, prayer and sex the least); but on average people were mentally absent for almost half their waking hours. Similar experiments in laboratories typically reveal lower levels of mind-wandering, but even here subjects are lost in thought around 25 per cent of the time. Far from carefully investing our attention, we wander mentally, and we don't even notice we're doing it.

Why do we squander attention in this way? The question is an important one, because by most common measures, mind-wandering is extremely bad for you. The fact that more than half of people in a 2012 survey of car crashes admitted to mind-wandering in the lead-up to the accident is only the most graphic evidence of its destructive powers. Mind-wandering at school or work makes it hard to learn or get anything done, and tests upon students in the US have confirmed a correlation with low SAT scores. Its emotional effects can be just as damaging, with studies repeatedly linking it to depression, anxiety and selfishness. A 2012 study that measured the length of chromosome tips in women to examine the impact of mind-wandering on ageing even connected it to shorter lifespans. In other words, having made your life a misery, your wandering mind is killing you.

None of this will come as a surprise to the growing band of mindfulness devotees, for whom being attentive to the here and now is the ultimate goal of sound mental health. When we mind-wander, we stop noticing what we are doing; by contrast, a mindful experience involves noticing everything to the fullest possible extent. Banishing mind-wandering is an explicit goal of mindfulness-based meditation, as former Buddhist monk Andy Puddicombe, founder of the hugely popular mindfulness app Headspace, made clear to the audience at TED in 2012. Citing that 2010 study by Killingsworth and Gilbert, Puddicombe said: "To spend almost half of our life lost in thought and potentially quite unhappy, I don't know, it just kind of seems tragic."

Tragic or not, in recent times mind-wandering has been undergoing something of a rehabilitation, first in psychological circles, and then more recently in the media. Research associating it with creativity and foresight has found coverage across the full media spectrum, from the Wall Street Journal (The Benefits of Mind-Wandering) to Stylist (How Procrastination Can Do Wonders for Your Career). It is early days yet, but mind-wandering is showing every sign of becoming a thing, buoyed to the surface of popular culture by the overlapping interests of business and self-help. At the root of this turnaround: the idea that mind-wandering is not a waste of attention but simply a different kind of focus.

Could this be the beginning of the revolt against mindfulness? The media reports do not make it seem likely. In their telling, mind-wandering is offered not as an alternative to mindfulness, but as a complement to it: "One mental mode is potentially just as beneficial as the other," as Fast Company puts it. A better question would be: why are these opposing philosophies of mind gaining popularity at the same time? What does it tell us about ourselves that we desire simultaneously to focus and escape? To answer this, it is first necessary to explore mind-wandering's scientific background.

The scientific interest in mind-wandering began with the mental equivalent of white noise. This was what psychologists thought they were witnessing in between the tests they were conducting on various parts of the brain. "Brain imaging was for a long time focused on which parts of the brain are active when we focus on specific tasks, such as reading words, recognising faces, or mentally rotating visual patterns," says Michael Corballis, emeritus professor of psychology at the University of Auckland and author of The Wandering Mind. "It was then observed that widespread areas of the brain were active between experimental trials."

Researchers realised that what had previously been dismissed as white noise was, in fact, purposeful neural activity. "The brain is active all the time," says Corballis. Even when nothing is happening, we still produce thought. Our internal life does not depend on external stimulation.

The areas of the brain responsible for these unprompted thoughts were grouped together in 2001 under the name of the "default network mode". Forget about left and right brains: the big distinction in neuroscience these days is between this enigmatic "default network" and the eager, hyperalert, "attention network", the part of the brain which responds to external stimuli. (They are, if you like, the brain's own cannabis and caffeine.) Mind-wandering is only one of many types of spontaneous thought the default network produces, but from a scientific point of view, its connection to the default network is the reason to study it. Understanding mind-wandering is one way to understand the brain's capacity to give birth to thought.

Mind-wandering is one of those paradoxical mental states, like sleep, which is impossible to enter into by conscious effort. To observe it, researchers adopt two main strategies: either they ask people to tell them when they have been mind-wandering; or they give them something to do, then interrupt them to ask if their mind is focused. (Examples of exercises subjects are asked to do include reading War and Peace, taking verbal reasoning tests and listening to Mozart.) Once it's been confirmed that a "task-unrelated thought", or TUT, has taken place, then psychologists can examine the neuroimagery to see what was happening in the brain at the time.

These studies give confirmation to all sorts of common intuitions. We mind-wander most when we are bored, or working at simple, repetitive tasks, such as washing the dishes; we mind-wander when stressed or tired; and when we are drunk we are more likely to mind-wander but less likely to realise we're doing it. By delving more deeply into the reasons we drift off, they also reveal the benefits of this peripatetic state, whereas previously the costs had dominated the discussion.

Chief among these benefits is creativity. The hidden dividend of zoning out – confirmed by tests on students with ADHD – is an increase in insight and imagination. "Mind-wandering is very unconstrained," says Nathan Spreng, assistant professor in the Laboratory of Brain and Cognition at Cornell University. "There are some cases where it's enormously beneficial to go for a walk and let your thoughts slow and have a chance to make connections… Let your brain just sort it out on its own." When we mind-wander we really can think of anything. This roaming allows us to forge the kind of associative connections that lie at the heart of creativity.

Mind-wandering also helps us think about the future. When we are focused on our immediate situation, our thinking tends to stray no further than the task at hand. When we allow ourselves to tune out, we can wander in time and space, reflecting on past events and contemplating possibilities for the future. "A lot of the mind-wandering that people engage in, the content of it is often future-oriented," says Spreng. "They're thinking about what they're going to be doing next, making connections about how to make that event come about." The exact reason for this prospective bias has not been established, but research suggests we use this material to prepare ourselves for the future, a process known as autobiographical planning. If, as many believe, the self is composed of narratives that tie together past and future, then mind-wandering may be where that storytelling is taking place.

In an age in which innovation is prized above all things, the combination of creativity and forward thinking is an extremely potent one. Mind-wandering has gone from relative obscurity to being a hot property. Yet it has not been cleared of its association with depression and anxiety, or of its detrimental effect on many essential functions. Hence the focus of current research: what makes the difference between positive and negative mind-wandering? How can we mind-wander effectively, instead of destructively?

Findings are already starting to emerge. One intriguing 2013 meta-study suggested that the key difference between healthy and unhealthy mind-wandering lay in what we were wandering from. "When we absent ourselves from a demanding task, we are much more likely to experience the negative effects of mind-wandering," the authors wrote. "By contrast, almost without exception, all of the adaptive features of self-generated thought are observed in less demanding situations."

This observation fits together with studies that show the best way of maintaining mental harmony during mind-wandering is to be able to be aware of the fact that you are doing it. Taken together, they present a picture of the ideal situation for mind-wandering: a state of calm monotony, neither too stimulating nor too boring, in a place where we are comfortable rather than frustrated. No wonder that, as study after study confirms, our best ideas really do come to us in the shower.

Yet just as this picture is starting to emerge, the conditions for positive mind-wandering are becoming harder to find. Constant intrusions make uninterrupted time a luxury, and instead of wandering in thought we are much more likely to find distraction online. "Screens are so information-rich they are not going to be stimulating the creative process in the same way," says Spreng. "Moving back gives our minds more freedom to explore the information already encoded in the brain." Like the rest of modern life, the internet is busy – it offers much, but in terms of mental nourishment, it is relatively barren.

In one strand of the literature on mind-wandering, it is possible to discern a strong critique of this precarious situation. Jonathan Smallwood, reader in the department of psychology at the University of York, and perhaps the foremost researcher in the field, has persistently attempted to emphasise that the content of mind-wandering is as important as the context: "It is not mind-wandering per se that we should be focused on when trying to improve health and wellbeing; rather if we want to be happy we should care what we mind-wander about."

In this view, the whole scope of thought seems to be taken into account: what we think, how and why we think it, when and where the thought takes place. A happy thought makes us happy – and where do happy thoughts come from but a genuine sense of happiness, of satisfaction with life and all it has to offer?

Will this approach take hold? From this perspective, what seems more likely is a Headspace for mind wandering: a snackable content delivery system designed to stimulate the default network, sold as a creativity aid to innovation-hungry companies. Work is already underway on such a solution. At a recent Central Saint Martin's showcase, textile designer Caroline Angiulo showed off a collection of "wander materials", modified versions of bubble wrap and tangled string, which employees in need of creativity could play with to induce a state of mind-wandering. Angiulo suggests the wander materials could be dispensed from office vending machines.

In similar fashion, a recent paper by leading researchers in mind-wandering concluded with the proposal that "future research should take advantage of advances in technology to help people to better recognise their mental states and adjust them accordingly to the situation". Eye-tracking software is currently used in trials to monitor subjects' attention – with business already experimenting with workplace surveillance, the combination of research and product seems inevitable. The name of this paper? It came with the title The Middle Way: Finding the Balance between Mindfulness and Mind-Wandering.

Why are mind-wandering and mindfulness attracting attention at the same time? The answer lies in our expectations of psychological self-help. Mindfulness is the perfect plug-in for late modern capitalism: it helps us cope while at the same time making us more productive. And although it might at first glance seem to be very different, mind-wandering has the potential to do the same. We are allowed to drift off, because, if we do it in the right way, then even those breaks are useful. It is okay to waste time, because wasted time is not in fact wasted at all.

This article was orginally published in The Long + Short digital magazine

Images courtesy of Shuttershock.com