he Harvard neuroscientist Baland Jalal is considering the unfathomable depths of the human brain. "It can contemplate itself, and vast galaxies, spirits, souls, poetry, Einstein. Yet you hold it in your hand and it seems so trivial." Jalal's passion – the subject that drove him to become a scientist – is sleep paralysis, a brain glitch that can fill its host with terror so palpable that it can cause lasting trauma. "What you get very close to in sleep paralysis," he says, "is that subjective world of the mind. It's very hard to put into words."

He should know – he's spent 271 pages trying to do so. His new book, The Phantom Mind, argues that by exploring the borderlands of sleep, where sleep paralysis occurs, we can delve deeper into how all aspects of our minds work. He may wear a lab coat and examine brain regions and neurochemicals, but studying the mechanics, he says, doesn't damp down the mysteries of consciousness. "For me, actually, I've become more spiritual and interested in the world of mystery and, I want to say supernatural, but dimensions that are intangible, by studying something like sleep paralysis."

He writes that up to one in five people have experienced these semi-waking, hyperreal visions at least once. Yet today, two people I spoke to about sleep paralysis had never heard of it. And it is most petrifying if it strikes before you know what it is.

I experienced it often during my late teens when there was no internet, or supportive Subreddit, to consult. My experience is fairly typical. I wake up, lying in my bed in my familiar bedroom with its comforting laundry smell, but I can't move. Worse still, something evil and malevolent is approaching the room – it feels like the moment of peak fear in a horror movie. Now it's in the room and all I can do is tense up and try to repel it with sheer will. But the monster approaches in a whirlwind that overwhelms my senses until it engulfs me and I am either dying, or being possessed. Eventually, I am able to move and the room is quiet again, but I struggle to believe it wasn't real. I feel as if I've witnessed something dark and supernatural. One night, it happened every time I started to fall back asleep and I ended up too scared to drift off again. When I told friends what had happened, it was impossible to convey how visceral it had felt. But I couldn't stop trying to make sense of it, or shake the feeling that it was real.

Jalal had similar experiences when he was a teenager. "I grew up in a ghetto-like neighbourhood in Copenhagen," he says. "Then one day, when I was 18 or 19, I was lying in my bed, and this ghost was trying to kill me. I was blown away by it." He wanted to tell his parents, but he knew they wouldn't believe him. "I thought, I have to figure out what's happening. This feels so bloody real. And so I decided to study the brain." He went to the American University in Cairo. "I was lucky to get scholarships and I studied this phenomenon day and night." Next he went to California, then Cambridge for his PhD and eventually Harvard.

It's not unusual for people to open their eyes during an episode and look around the room. One girl he tells me about could see herself in the mirror in her room. "Lying there, obviously she couldn't move, but she literally saw herself turning into a monstrous creature."

You feel as if you are awake when it happens, but ultimately you then "wake up" from it, so it's hard to know whether you're fully awake or a little bit asleep. In truth, a glitch occurs somewhere in between. When you dream, you're physically restrained for your own safety. "In REM sleep, you're completely paralysed from head to toe. This happens in the brainstem," says Jalal. "Occasionally, as the sensory cortex, the perceptual part of the brain, awakens, you have this wakefulness emerging and mixing with the brain. You might say sometimes you feel very much awake and aware of your sensory surroundings. And sometimes it's more blurry. The boundary between the dream world and the waking world is very blurry."

Your level of wakefulness will depend on the level of prefrontal activation – the prefrontal cortex being where high-level thinking and awareness happens. "That can vary a lot from person to person, and between one sleep paralysis episode and another."

The presence, which is often reported as sitting on the host's chest and suffocating them, arises, it is thought, from the distressed brain trying to understand why the body seems to be held down. "When you are paralysed, and you can't speak, your thoughts will go to a potential predator, somebody trying to hurt you. You feel very vulnerable," says Jalal. Brains are storytellers, he says. They join the dots to tell us a story of what's going on based on the information available.

He has some possible explanations for why so many people see a being of some sort. "We think that some of the systems that have to do with a sense of body image get distorted, and so for that reason, you tend to hallucinate bodily figures. You will feel like there's a ghost in the room with you." But the ghost, in a sense, is your brain needing to find its body. Sometimes, instead of a monster, there's a doppelganger. "I've had that before," he says.

It was more bemusing than scary. "I felt as if my self was being hijacked, and I was having this conversation with myself about who was the real me, the lad up there or the one down here. So your sense of agency gets muddled." In sleep paralysis, "when you are lying down, unable to move and speak, your sense of self gets distorted, mainly because as you try to move and escape the paralysis, there's all this misfiring going on in the brain, affecting the body image structures of the brain … then you hallucinate these figures."

How scary it is depends, he says, on the extent of activation of the limbic system, specifically the amygdala. "If you turn up the volume on the amygdala, a little almond-like structure and the fear part of the brain, the fear would probably start to amplify, and then my Baland out there would probably morph into a demonic creature. So it really depends on the emotional state that we are in. That can taint and colour our visual percepts and hallucinations."

Anxiety, he says, can trigger sleep paralysis, and make it more frequent. "We see a lot of people with anxiety disorders that have it. The condition seems to exacerbate anxiety and even trigger post-traumatic stress disorder (PTSD) in some cases." He cites a 2004 study where participants had their stress responses measured as they were listening to audio tapes of themselves explaining particularly monstrous episodes of sleep paralysis. "As they're listening, their heart rate is going up. Their sweating is profuse." When the team compared these reactions to people who heard audio tapes of their own PTSD trauma stories, they were comparable.

As Jalal says himself, the monster feels real. "That is the very nature of sleep paralysis which makes it so bloody intriguing." This is because, he thinks, the monsters are hallucinations as opposed to dreaming. "This is a true hallucination. You feel awake, and yet you see something."

He puts this down to the same receptors – "which are doorways by which cells can communicate" – that psychedelic drugs bind to. Namely, receptors for "a type of serotonin chemical called serotonin 2A, which is responsible for that sense of overwhelming dread, fear and horror. Once you activate those, the fear will become overwhelming, and your brain will need a source to latch on to and say: 'Oh, this is coming from here.'" In a sense, it's like a very bad trip. The vividness, he supposes, comes from a mixture of this, plus feeling very much awake.

When I've had sleep paralysis more recently, I try to shake myself or hum so my sleeping partner might notice and wake me up. Occasionally, it has worked. Sometimes, says Jalal, "you can actually jolt yourself out of it. Some people will wiggle a toe or a finger or something, and they can escape from it." This can backfire, however. "As you try to move, often what happens is that it comes back with a vengeance, and you feel pressure on your chest, your legs, your arms feel as if they are spasming, and it brings on the panic. It's this real cocktail of horror."

Cultural trends play a large part in shaping the hallucinations, which feels disappointing, a bit like doing a Ouija board, and even though you swear you're not pushing the pointer, it only spells out what you're thinking. The unconscious mind, he says, will jump in and flesh out the monsters. The "hat man" is a common one, which researchers liken to Freddy Krueger. "If you're into sci-fi, then you see an alien. If you live in Egypt, then it's a jinn, an evil genie." Mine felt more like the mind flayer from Stranger Things, but that didn't exist back then.

Stories surrounding sleep paralysis are particularly embedded in some countries, such as Italy and Egypt, which results in it happening more. "Your grandmother might say: 'There's this monster that comes at night. It looks like this.'" Lo and behold, it happens. Then it can become a loop, he says. You become scared of sleeping. You might have it two days later again; it might become chronic. "Perhaps you start telling other people too, and it spreads. In Egypt, because of the cultural ideas, people would have it more and for much longer episodes."

Jalal has had people tell him that an attack lasted up to an hour, but he says they can't know for sure, and it sounds "highly implausible". They don't often get a chance to record episodes in a lab but, "the ones that we've seen usually last a few seconds to a few minutes, and then your brain pops back. It can't stay in that state of limbo. It's a very unnatural state, and eventually your sensory cortex will align and fall back in to place. But it can feel like a very long time." Mine only felt like a few horrific minutes.

He has been working on a strategy to cut the anxious feedback loop in those with chronic sleep paralysis. The first step is closing the eyes to minimise any kind of stimuli from the room that can affect hallucinations. Then the idea is to reframe the situation as a common, harmless occurrence and try to become emotionally detached from it. "Worrying won't change anything." Then focus attention on something positive and comforting and don't try to resist the paralysis and move – accept the situation. While it only occasionally worked on him, a small pilot study in Italy with narcolepsy patients (about half of whom apparently get sleep paralysis), found "about 50% reduction in sleep paralysis in people who had it very frequently. They weren't as anxious about it, so it didn't have the same priming effect."

Jalal would love to be able to do more experiments testing how these experiences can affect people's beliefs. "I'm a religious scientist, which is rare," he says. But he once had a psychologist who was a "stern atheist" call him after experiencing sleep paralysis and say: "I'm changing my mind now. It feels like there's a gateway opening to something beyond the material world." It wasn't the first time Jalal had been told this. Non-believers, he says, can "become more open to something beyond their physical realm. So perhaps we can find what you might call 'the spirituality circuit' in the brain, by scanning people during sleep paralysis." From where he's standing at least, "there's something cosmic going on."

The Phantom Mind: Insights from the Borderlands of Sleep by Baland Jalal is published by Allen Lane (£25) on 12 November. To support the Guardian and the Observer, order your copy at guardianbookshop.com