听
By Shannon Mullane (MJour鈥�19)
You鈥檙e flying over a luscious landscape filled with clouds, streams and blossoming plants. You turn your head, and an eagle comes to join you. Music rises and falls depending which way you look, how fast you move.
Then, later, you remove the virtual reality goggles, and you鈥檙e back in the hospital. One more physical therapy session down.
That鈥檚 the basic idea for a new virtual reality experience designed specifically for the health care setting鈥攁n experience created in part by CMCI lecturer Sean Winters.听
Winters, who works in the Department of Critical Media Practices, has spent years investigating the countless relationships that exist between sound and images. In 2019, he partnered with Reality Garage, a VR lounge and maker lab in Boulder, and started using his audio expertise to help young patients undergoing treatment at a Colorado hospital.
It has been one of the biggest, hardest and most complicated projects he鈥檚 ever undertaken, Winters said鈥攁nd it has left him inspired to do more.
鈥淚t鈥檚 extremely challenging . . . The stakes are real. The patients are real,鈥� he said. 鈥淚鈥檝e had to utilize so many of the skills I鈥檝e learned through the years.鈥�
The VR experience is designed for patients undergoing a type of physical therapy in which electrical currents help muscles contract. The therapy can help strengthen muscles, address stiffness, improve balance and more, according to health care organizations like the National Multiple Sclerosis Society. Health professionals use it as a treatment for conditions like multiple sclerosis, strokes, spinal cord injuries and nerve damage.
Real-time audio synthesis in virtual reality鈥攊t鈥檚 pretty serious business.鈥�
Sean Winters
DCMP Lecturer
The therapy can be challenging for patients, especially if they have to repeat it for long durations several times each week. In the hospital, patients will don VR goggles and noise-canceling headphones on a bicycle-like apparatus. As they pedal at different speeds to improve the use of their muscles, the experience will allow them to fly.
鈥淭he therapy can be oppressive, and boring, and just not exciting,鈥� Winters said. 鈥淥n the one hand, we鈥檙e simply trying to create an experience that鈥檚 fun, entertaining, looks cool, and is replayable. . . . But then at the same time, we spent a lot of time crafting this whole narrative that鈥檚 based around healing and growth.鈥�
The designers, however, faced one initial hitch: Typically, users control VR experiences with the help of a hand-held remote鈥攚hich isn鈥檛 an option during this type of therapy.
That鈥檚 where Winters鈥� audio expertise became crucial.
Winters and the Reality Garage team decided to make the program audio-driven and visually navigated. As users fly听through the water-rich, lush biome, their gaze activates different game elements, like collectibles and power sites. But sound听guides the user just as much as the visuals.听
Some audio elements are 2D: No matter which way users turn their heads, the sound stays the same. Then there is immersive spatial audio, a 3D element. When users pass something, its sound is only heard in one ear鈥攖hen both ears after they turn their head to look at it.
Because patients need to meet certain rotations per minute on the bicycle for their treatment, rhythm is vital. Winters made sure the beats per minute of the audio would match the exercise program: slower for warmups and cool downs, faster for higher intensity pedaling.
But one of the most innovative elements of the program involves real-time data. The therapeutic goal is for patients to hit a target speed. If patients are pedaling slowly, the real-time data will inform the program, and they鈥檒l hear only half of the sound frequencies. Once they start pedaling faster, they can hear both halves of the听frequency spectrum, Winters said.听
鈥淩eal-time audio synthesis in virtual reality鈥攊t鈥檚 pretty serious business,鈥� he said.
This year, the design, testing and final stages of the project are winding down. As Winters reflected on the project鈥檚 progress, he said the experience has tied in everything he has studied, practiced and explored in classrooms with students.
Looking ahead, he envisions more virtual and augmented reality programs in health care settings鈥攊t鈥檚 an intuitive, logical and very exciting prospect, he said.
鈥淭hinking about patients in hospitals is extremely intense, and I can only begin to imagine what that experience is like,鈥� Winters said. 鈥淭o bring even just a tiny, tiny amount of positivity or joy to that situation is so, so thoroughly inspiring and motivating to me.鈥�