Virtual Reality Transforming Brain Surgery for Patients and Providers

A journey into the brain for patients and families

Emri Bullock was 11 years old when Auguste performed her first brain surgery at UCSF Benioff Children's Hospital Oakland. At age 4, she had been diagnosed with neurofibromatosis type 1 (NF1), a progressive genetic condition that primarily affects the skin, nervous system and eyes. Symptoms of NF1 range from mild to severe. For Bullock, the first tumor appeared in her brain.

Auguste removed the tumor, which was the size of a golf ball, in 2014, but it recurred four years later. Two more tumor-removal surgeries appeared to be successful but the areas later developed cancerous tumors, discovered on her quarterly MRI checkups. Emri consequently received intensive chemotherapy, missing several months of high school.

"Each reoccurrence was frustrating, but Dr. Auguste kept me positive, and I knew I was in good hands," said Emri, who is now 21 years old.

For Emri's mother, Jackie Bullock, having VR technology at the start of her daughter's surgeries would have made a world of difference.

"VR technology was not available when Emri first began treatment for her brain tumors," Jackie said. "If it had been, it may have considerably reduced the stress of the surgeries. When Dr. Auguste led Emri and me on a flight of Emri's brain following her last brain surgery, I felt included and relaxed because I could see it for myself and was confident that he would know what to expect if a similar tumor reoccurred."

Emri found the VR flight jaw-dropping. Following avatars of her parents and herself on a VR journey through her own brain – guided by an avatar of Auguste in a white lab coat – was unreal, she said.

"I took flights of my brain after my last surgery and couldn't believe how big the mass was," she said. "At the end, I said 'Wow, that was in my head?' But it felt great to see it and see for myself that it was gone."

For Auguste, VR helps him communicate with his patients in a venue they can relate to.

"These kids are well-schooled in video games, so they light up at wearing goggles for a flight into their own brain," Auguste said. "Seeing a physical representation of the lesion that has been causing their symptoms is very empowering. It is also vivid, colorful and visually engaging, which they love."

X-ray vision through augmented reality

As technology evolves, so does this surgeon's approach to his craft. While VR offered a computer-generated simulation of an alternate world – in this case, a representation of a patient's brain that Auguste could enter and explore to chart the safest surgical path – he has embarked on the technology's next generation by adopting augmented reality (AR) in his operating rooms.

AR blends the virtual content created from 2D images with the real world by superimposing holograms on the subject, which is then viewed with AR glasses or a mobile phone.

The result, he said, allows him to visualize a surgery with more confidence and gives him greater precision in removing abnormal tissue while preserving, avoiding and protecting the rest of the brain.

"With AR, 3D holograms are superimposed onto the patient's brain while I operate under the microscope," Auguste said. "The effect is like having X-ray vision. I'm seeing the tumor that is beneath the surface, so I can pinpoint exactly where I need to make incisions."