A “Remapped Reality” – Using Virtual Reality to Remap the Visual

On Tech & Vision with Dr. Cal Roberts Podcast

Today’s big idea is the power of virtual reality — how people are using VR to remap sight and help people with vision loss. Dr. Roberts visits with Grace Andruszkiewicz and Dr. Frank Werblin about how emerging technologies help people with low vision access the areas of vision they still have, by using a virtual reality system. They also talk about how sight works biologically, and how one such device, IrisVision, works to connect people socially.

Episode 3: A “Remapped Reality” – Using Virtual Reality to Remap the Visual

 Wow, this is amazing.

Roberts

Have you ever tried virtual reality?

This is so cool.

Roberts

Or better yet, have you been with someone trying virtual reality for the first time?

I’m in this dinosaur world.  That’s so cool.

Roberts

If you’ve tried VR, you know how eye-opening it can be.  Without leaving the comfort of your home you’re transported to a new world.  It’s revolutionary for entertainment, for education, and even more so for people who have physical limitations.

Mickey

…It’s coming!  This is absolutely unbelievable!

Roberts

This is Mickey who was aphasic.  Totally non-verbal after a diagnosis of dementia.  Here she is using a VR headset that has placed her in a roomful of puppies, and as you can hear, she’s responsive.

Mickey

Oh!  Look at that.  I can almost feel it.

Roberts

Rendever, a Boston-based company, leverages the power of virtual reality to help older people to overcome social isolation.

Andruszkiewicz

We primarily work with senior living communities.  And these folks move in, and they’ve lived their whole lives, built their families, have had hobbies and jobs.  All of a sudden they’re moving into a one-room apartment in a community.

Roberts

Grace Andruszkiewicz is Director of Marketing and Partnerships at Rendever.

Andruszkiewicz

They don’t know anyone and their sense of freedom really gets limited, like overnight.  And so, the ability to use a technology like virtual reality to allow them to check off bucket list items, travel to places that have been on their list forever.

You want to reach out and touch them.

Wow, I know.

What are you seeing?

Dolphins all around us, swimming.

They’re having a great time swimming.  Whoa!  They’re so big, those two.

Andruszkiewicz

What’s really meaningful, too, is helping people go back to places that are really emotionally meaningful from their past.  So being able to stand in front of the house that you raised your kids in, or the church that you got married in or the school you went to, or where you went on vacation every summer for your whole life.  Those special places to people really help them open up and what happens, really organically, is people start telling their story.

You get to hear all of these memories from the past and emotions about how these places made you feel.  Or often it’s activities, too.  Maybe somebody was an avid hiker or biker.  When they feel like they’re back in that place or doing that thing, they come alive again.

Roberts

This is Lighthouse Guild’s podcast, On Tech & Vision with Dr. Cal Roberts, where I talk with people with big ideas about how technology can make life better for people with vision loss.

I’m Cal Roberts, and today’s big idea is using the concept of VR to remap sight to help people with low vision come back to their lives.

Whether you’ve experienced VR or not, you probably know VR is a computer generated simulation that uses special devices like goggles with a screen, or even gloves fitted with sensors to change the sensory inputs your brain receives.  The visual field your eyes are scanning.  The sounds you’re hearing.

Andruszkiewicz

They put on the headset.  What happens is they are immediately in what we call our waiting room, which looks like a back porch, a really beautiful, modern back porch with a fireplace and butterflies.  People often think that that’s the whole thing.  So cool and so pretty.

Roberts

By changing these sensory inputs, VR tricks your brain into believing you’re in another reality.

Andruszkiewicz

We have a huge library of video experiences, which is kind of the traditional VR experience where they’re all kind of categorized into adventure, nature, animals, music, education, religion, etc.

Roberts

No longer in your living room, or, for Rendever’s users, no longer in your senior living facility, Grace shared some anecdotal evidence of the device’s impact that was pretty personal for her.

Andruszkiewicz

I joined the company just over a year ago and I had done some consulting with the company prior to that.  Before I took a full time job I asked if I could take one of our demo systems to my own grandmother and see what her reaction was.  From my perspective, if she’s not into it, I’m not into it.

And so, I brought the system to my grandma.  She lives independently and she’s 90, and she’s 90 pounds soaking wet, and is just the most amazing woman.  We got to go back to her childhood home, which I had never seen before.  She was like, oh, if we cross that little creek and take a left we’ll be at my church.  So we walked down the street and were able to go see the church that she grew up going to.  It was a really special moment.

Roberts

At Lighthouse Guild, we’re focused on people with vision loss.  We asked Grace if virtual reality, which relies so heavily on video, has a place for people with vision loss.

Andruszkiewicz

Anecdotally, we have heard so many stories of people that have some sort of vision impairment.  Some sort of macular degeneration that have not seen anything clearly for years.  Then they put on a VR headset, and because the screen is 1/2 inch from your eyes, it’s not uncommon for people to see something clearly for the first time in a VR headset.  And that has made people cry.  The ability to have your sight back, even just for 30 minutes a day is something that is so moving to see happen.

Roberts

It’s amazing.  But what if a VR device, rather than presenting a virtual environment, modulated a low vision person’s visual field so they could see better.  This is the brainchild of our guest today, Dr. Frank Werblin, Founder and Chief Scientist of IrisVision.  He spent his career in visual neuroscience, studying retinas to restore lost vision.  When he discovered the progress being made in VR, a lightbulb went on.

How did you come to the realization that a device that was meant for gamers could be something that could benefit those with low vision?

Werblin

I had been following the development of virtual reality from early on which was promoted by a young kid in his garage named Palmer Luckey in Southern California.  It was a fascinating development because he was able to produce a low-cost virtual reality system that revolutionized that world.

I realized that the kind of processing that he was doing could be implemented to help people with low vision.  It’s relatively straight forward.  What’s needed is a processor that has the capability of remapping the visual world.  And that’s what I was able to do.

Roberts

Would you characterize what IrisVision does as virtual reality?

Werblin

Well, it’s not really.  We use a virtual reality system.  The system that was originally designed for virtual reality.  It’s a Samsung Gear headset, and the Samsung phone.  They developed that together with Oculus, which was a virtual reality company in Southern California.

We hijacked that technology, sort of take it’s guts out and reconfigure it so that it’s useful for low vision.  In fact, it’s hard to describe what reality we’re talking about.  It’s not augmented reality either.  It’s sort of real reality.  Or remapped reality.  Something like that I guess you’d call it.

Roberts

Just to clarify, the device is for people with low vision – people who have some remaining vision.

Werblin

There’s some visual capability left, but most of these people live a life as though they’re blind, because they’re not able to utilize those islands of sight that remain in the vast sea of emptiness in their visual world.

It occurred to me that what was needed was a low cost, non-invasive device.  Something that people could take on and off that could recode the visual message in a way that would resonate with those islands of sight that people still had remaining.  So what we do is remap the visual world that they’re seeing so that it becomes kind of resonant with their pathology.  As a result, people who have been acting in a very blind way are brought back to life, because for the first time they can recognize faces.  It’s a wonderful thing.

Roberts

So, how does the device work?  How does IrisVision remap reality?

Werblin

The idea is that different people with different low vision pathologies have different requirements.  People with macular degeneration use a magnifying glass so that they can hold the magnifying glass up and read text, or a read a pill bottle.  Something like that.  The big problem is that they can only magnify in a region that extends as far as their arms.  They can’t magnify to see something across the rooms.  They can’t magnify to see items on a supermarket shelf.  They can magnify to see their grandchildren sitting at a table across the room from them.

This gives them the ability to magnify at a distance.  That’s the great value of it in that way.  And it leads to social connection, which is the thing that they’re most lacking because they’ve been seeing their friends or their children.  They can interact with the expressions on their faces.

Roberts

You started by designing to address vision loss associated with macular degeneration.  Are there other examples of visual impairment that IrisVision can help with?

Werblin

When I started out, I only had a solution for macular degeneration.  It’s hard to believe, but it took me a year and a half and one day wake up and say, look, I have access to the images to both eyes.  I can reconfigure those in any way I want.  I should be able to resolve any kind of visual pathology.

So the next one I tackled was retinitis pigmentosa, which is quite the opposite of macular degeneration.  People with retinitis pigmentosa, call it RP, have tunnel vision.  It’s like looking at the world literally through a soda straw.  They just have that tiny visual field right in the middle of everything.  As a result, it’s difficult for them to integrate that soda straw across the whole 180 degree field of view.

People with tunnel vision that have difficulty scanning a broad field from one ear to the other if you look across the world like that.  So, it occurred to me that it might be useful to make their visual world much smaller.  Instead of magnifying the world, I minified it.

Roberts

Where did this idea come from?

Werblin

It came about almost by accident.  I was talking one day to one of our investors who has RP.  He said to me, the only time I can see myself is when I look at my reflection in the bathroom faucet.  The bathroom faucet is chrome, and it’s convex.  So it forms a lens.  It’s a lens that minifies things.  So I thought, a-ha!  This guy is onto something here.

So I decided to try that and see if it works.  And it did.  To give you an example, right after I started doing this, we had a patient, a woman who came into our office who had a 1 degree visual field.  Do you know how narrow that is?  It’s like holding your hand out, arms length, and looking at your thumb.  Your thumb spans about one degree.  Imagine that’s all you could see.

What we did was to shrink the visual world down from 180 degrees for her to 30 degrees.  With that one degree field, she could scan 30 degrees very easily.  For the first time in maybe 20 years she was able to see her husband.  So, you know, everybody started to cry.  This was quite amazing.

Roberts

And this minification of the world, this modification of visual reality, was possible because of the technology of the Samsung phone and the headset originally designed for VR.

Werblin

I didn’t invent minification.  It’s existed before and there are a number of academic studies describing it.  But there are very complex forms of minification.  And if we use those complex forms of minifications the battery life of our system would be five minutes.  So I really simplified the thing tremendously so that we just have this smaller version of things.  And it’s amazing how effective it can be.

Roberts

So, you woke up one day and thought, through this phone I have access to everything that both eyes would see if they had complete vision.  And that was sort of a lightbulb moment for the modification.  You said there were 12 or so other ways that you manipulate the visual world?

Werblin

Yeah, depending upon their visual disfunction.  So much that can be done.  Some of it is retinal, and those are the two I described.  Some are more cortical.  There are cortical pathologies that we can also address this way.

Roberts

The retina is the light sensitive layer at the back of the eyeball that triggers neuron transmissions to the brain.  And the visual cortex is the part of your brain that processes visual information.  Dr. Werblin, could you quickly give us an example of what the technology does differently for a retinal pathology versus a cortical pathology?

Werblin

It’s going to be a different algorithm, but using the same technology.  I’ll give you an example.  One of the things we can look at is hemianopia, which is a cortical disease where you lose the ability to see one half of the visual world.  It’s not like one eye is closed.  You’re using both eyes, everything on, let’s say the left side of the visual world, is missing.  When that happens people begin to ignore the left side of the world.  They’ll walk into doors or fire hydrants.

The technology there is to inject a picture of the missing side of the world into the sighted side of the world from time to time to remind them that the rest of the world is there.  That’s another example of a way to deal with different sort of visual disfunction.

Roberts

So that’s how the device works.  It’s not virtual reality, but it uses the premise of virtual reality that the computer processor and the camera in a phone mounted on a headset can redefine the user’s visual field and change a person’s experience of their reality.

At Lighthouse Guild we’re all about helping people with vision loss lead full lives.  How can IrisVision help people lead fuller lives?

Werblin

Correcting for or assisting patients with visual loss has a much broader function.  And that is, it reconnects people with each other.  It’s a social function.  IrisVision assists people in seeing the visual world.  But what it’s really doing is reconnecting them socially.

Roberts

I imagine development requires a lot of user testing.  Can you think of an early example where you shared the product with a person with vision loss.

Werblin

I can.  One of my first users was an 84 year-old woman named Liz Raymer, who was a friend of a friend.  I was very lucky to find her because she was extremely courageous and willing to work with me to sort out the bugs in the system.  She was able for the first time to make contact with her grandchildren.  She could play cards with them, see the cards, then look across the room and see the expressions on the kids faces.

Although this is a visual assistive tool, it’s primary value is in establishing and maintaining social connection.  The thrill of seeing a child smile for the first time is overwhelming.  Remarkably, Liz also took the device to Europe one summer and toured museums.  The Picasso Museum in Paris.  The Van Gogh Museum in Amsterdam.

Roberts

Where are you going from here?  What’s next for IrisVision?

Werblin

Here I’m working with a phone in a headset.  And the phone, just because of the way it’s manufactured, designed, is internet connected.  It occurs to me that if that’s the case, maybe people wearing this headset could communicate with their ophthalmologist over the internet.  They don’t have to come into the office.  We’ve been developing a set of visual tests that can be implemented on the phone.

Now the patient, staying at home, can be visually tested, can be analyzed, and can be assisted by a central clinician.

Roberts

Where are you in the process?  You said you have the tests created.

Werblin

Yeah, we have the tests created.  They haven’t quite been deployed yet, but they exist and they will be very soon.

Roberts

It’s interesting to note that Dr. Werblin started his career at the intersection of electronics and biology.  He studied engineering at MIT before he became a graduate student at Johns Hopkins University.

Werblin

I was a graduate student at Johns Hopkins looking for a thesis topic.  One day I went to a lecture given by young professor named John Dowling.  A young assistant professor.  And he had just worked out the wiring diagram for the retina.  So here’s this guy talking about a wiring diagram and I thought, hmm, there’s something in here for me.

When the lecture was over I went up to him and said, it’s nice to have the wiring diagram, but what are the neurons doing?  He said, well, nobody knows what the neurons are doing.  Why don’t you take it on as a study?  I thought he was just being kind of patronizing.  I said, oh yeah, yeah.  Sounds really good.  I went home and I wasn’t really thinking about it much anymore.  About a week later I got a call from John Dowling.  He said, well, I’ve set up a lab.  You can start doing your experiments now.  This was the most amazing stroke of luck anybody could have possibly had.

Then I began experimenting and everything I did for the first couple of months was wrong.  Every mistake anyone could possibly make.  But over time I developed the technology to understand what all the retinal neurons were doing.  For the first time we had a catalog of what they were doing and from John we knew how they were wired together.  So we had a functional description of retinal processing.

Roberts

For our listeners, can you explain how the brain and the eye are connected?  Can you explain the process of sight?

Werblin

Well, I can only explain it in the simplest terms.  It all begins the retina, which organizes and codes a visual message.  It’s very complex.  Just to tell you some things about retinas, for example, this is a tissue that’s about the thickness of a piece of tissue and it’s squishy.  It’s malleable.  But when you get inside and look at the details, at the anatomy, for example, it is unbelievably sophisticated.

In the inner part of the retina there’s a region where there are layers of neuro processes.  Each later is a couple of microns thick.  But there are at least 20 of them.  What’s remarkable about this is that each of those layers encodes a different quality of the visual world.  Some encode edges.  Some encode motion.  Some encode color.  There are 20 of them, at least.  And the images, these extracted images from each of these layers travel through the optic nerve to the brain.

Roberts

The images travel, or is it just like an electrical signal, like an on off computer type of thing?

Werblin

It’s images, because they exist in two dimensions in space and time, but they’re neural images.  If you could somehow extract the information that’s carried in neural signals and make them look visible, you would see 20 different extractions of the visual world.  These travel to higher centers in the brain.  Without going into all of that, I have the same question that you do, which is – who’s reading these out?  Who’s looking at all of these images and putting together meaning in the visual world?  And I think that’s a great mystery.  It’s not solved.  It bears on the issue of consciousness.  And who knows anything about that.

Roberts

There’s a lot we don’t know about what happens in the brain when someone sees.  But what do we know about what happens in the brain when someone uses an IrisVision device?

Werblin

You’ve got to provide the brains with the raw material.  I think the remapping that we do provides access to the raw material for the brain.  It comes through the retina.  Because there is something really fascinating happening in the brain, which is that after people use this device for a few months, they take it off and they discover that their native vision has improved.

Roberts

That is remarkable.  Is it a wide-spread phenomenon across users?

Werblin

We don’t know yet.  It excites me because I’ve seen it in so many people, but we haven’t formally studied it.  There’s something that’s well-known in neuroscience that’s related to this which is brain plasticity.  That is, when you use parts of your brain more, they become more efficient and more effective in order to use.  I think that’s what’s happening here, that we’re inducing a lot more visual activity because people now want to get out and see things.  They’re not sitting passively in the corner with no friends.  They’re out interacting with the world visually.  And I think what’s happening is that we’re waking up neurocircuits that had been dormant.

I don’t have proof of that.  I want to study it.  It’s such a dramatic observation that I made that it must be something like that happening.  So, the brain is actively involved in learning how to deal with the remapped images.

Roberts

As a graduate student, Dr. Werblin, you did this research to understand how the retina works by relying on the emerging technologies of the time.  Field effect transistors and integrated circuits.

Werblin

If you use existing equipment to do experiments you can only solve older problems, because that equipment evolved from older problems.  If you want to solve new problems, you need new technology.

Roberts

Which brings us back to how you’re using the emerging VR technology for the IrisVision headset.

Werblin

So far we’ve been really lucky.  We haven’t manufactured anything.  We can just take the phones and headsets off the shelves.  As we move ahead we’re going to start getting into manufacturing because we’d like to make the headset a lot more attractive, a lot lighter, a lot more user friendly.  And that doesn’t exist.  But we could make headsets like that.

It won’t always be black like that, either.  I think a smaller white one would be much more attractive, much more accepting to the community.

Andruszkiewicz

I think there’s a misconception that VR is primarily used for gaming, though we get lots of raises of the eyebrow when we talk about older adults and VR.  I think when people think about the technology, they automatically imagine teenagers isolated in their rooms gaming for hours on end in the middle of the night.  We’re doing the exact opposite.  We’re putting these headsets on older adults in a group setting to build community.

While the technology can be used in so many different applications, the ability to think about it in a way that can serve humanity is something so special.

Look at that!  I can almost feel it!

Andruszkiewicz

For us it was really a no-brainer.  This ability to help people to transport to different places and participate in activities that they otherwise wouldn’t be able to.  For us, it’s just not older adults.  It’s anyone whose life is limited in anyway.

Roberts

Like VR headsets do for the sighted gamers, or for Rendever’s mostly-sighted clientele, the IrisVision changes the game for people with low vision.

Bill Graham is on the technology advisory board at Lighthouse Guild and walked us through how his IrisVision device transports him to new places.

Graham

I had it now for two years.  It was introduced to me at the Lighthouse by my doctor.  I just recently got an exercise bicycle, and it has a screen, but I can’t see the screen unless I use it.  So to set it up I have to get my thing and I can take trips through Rome, Athens, wherever.  But I need my IrisVision to see the tour I’m going to take.

For things like that it’s the only thing I could use.  Without that I just wouldn’t be able to do it.

Roberts

So, even though the IrisVision headset doesn’t technology offer a virtual reality experience, it does bring users to new worlds, and it offers new opportunities for people with vision loss who can have more independence than they would without the device.

This has been Lighthouse Guild’s On Tech & Vision with Dr. Cal Roberts.  In this episode we explored how emerging technologies like smart phones, super fast processors, tiny cameras, and virtual reality headsets can help people with low vision to access the areas of vision they still have.

We talked about how sight works biologically, and we examined the IrisVision headset, one device that uses some of the basic principles of virtual reality.

We exist on the forefront of a tech revolution.  At Lighthouse Guild, we believe creative entrepreneurs and developers – maybe you  – will build on these advancements to deliver solutions for people who are visually impaired.  Or a person with vision loss?  Did this episode spark ideas for you?  Let us know at podcasts@lighthouseguild.org

And if you like this episode, please subscribe, rate and review us on Apple Podcasts or wherever you get your podcasts.

I’m Dr. Cal Roberts. On Tech & Vision is produced at Lighthouse Guild by my colleagues Cathleen Wirts, Jaine Schmidt and Annemarie O’Hearn.  My thanks to Podfly for their production support.  For more information, please visit www.lighthouseguild.org.