Training the Brain: Sensory Substitution

On Tech & Vision Podcast with Dr. Cal Roberts

Today’s big idea centers on the place where big ideas get born — the human brain. In today’s episode, Dr. Roberts and his guests explore theories of brain plasticity, sensory substitution, and sensory augmentation. Dr. Patricia Grant discusses the BrainPort, which uses sensory substitution in this case, the nerve fibers in the tongue, to send information to the brain instead of the optic nerve. Dr. John-Ross Rizzo is developing a device to be called the Sensory Halo, which is supported by sensory augmentation. Both guests share what is being learned about sensory substitution and augmentation through these technologies and how this understanding will help perfect future devices to enable people with vision impairment to see better.

Podcast Transcription

Rizzo

And I kind of went into a funk for a while.

Roberts

When he was about 14 years old, after years of vision problems that started in his childhood, Dr. JohnRoss Rizzo was diagnosed with choroideremia.  Choroideremia is a retina condition, characterized by progressive vision loss.  It usually starts with night blindness, then progresses to tunnel vision, then to a loss of visual acuity.

Rizzo

It took me about a year of sour grapes and I realized that I had this kind of penchant for science and I’d always done well and I just decided that I would focus on eye care and figured medicine was it.

I think, even to my doctor’s surprise, I ended up going to NYU on scholarship and was actually admitted to medical school on scholarship, and one of my doctors at the time during a checkup said, you know, you’re never going to be able to do what an optometrist is going to need to do.

And that was hard.  That was a tough couple of weeks.

I wasn’t really sure what to do, because I had really set my sights on ophthalmology.  I thought I’d be an ophthalmologist.

Roberts

Weeks, he said.  Weeks!  While he was stunned and saddened by the realization that this condition ended his dream of becoming an ophthalmologist, the young JR didn’t give up.

Rizzo

I found out about disability medicine, or rehabilitation medicine.  I fell in love with it and the rest was history.

Roberts:  He did his chief year of medical residency at NYU, studying disability medicine.  And he noticed something.

Rizzo

I realized that while we did a lot of amazing things for disability – physical impairments and different sensory disabilities, we didn’t do a whole lot for visual disability specifically.  I started applying for some grants to think about clinical research.  I figured I’d do a pivot again.  So, pivots have kind of been part of my progression between life chapters.

Roberts

The entrepreneurs we talk to on this show say the same about building a business or launching a product.  It’s all about testing.  Getting feedback and iterating on your original idea.  Pivoting worked for JR Rizzo.  Today Dr. Rizzo is a neuroscientist and assistant professor at NYU’s Departments of Rehabilitation Medicine and Neurology as well as at NYU’s tandem School of Engineering in biomedical, mechanical and aerospace engineering.

He is applying his expertise to building a sensory augmentation device for people with vision impairment that we’ll hear more about later in the show.

Today’s big idea is truly big and it centers on the place where big ideas get born:  the human brain.  Today we’ll explore theories of brain plasticity, sensory substitution and sensory augmentation and the roles these will play in future technologies for people who are visually impaired.

Our guests today are Dr. Rizzo, whom you’ve met, and Dr. Patricia Grant, the Director of Clinical Research at Wicab, the company that made a device called BrainPort, as well as Director of Research at the Chicago Lighthouse.  Dr. Grant has made sensory substitution a key focus of her work.

Grant

Sensory substitution was actually introduced in the 1960s by Dr. Paul Bach-y-Rita, who is also the developer of the BrainPort.  And he is also known as the father of sensory substitution.

Dr. Bach-y-Rita was a neuroscientist.  His father Pedro suffered a stroke.

Roberts

This was in 1959 that Pedro Bach-y-Rita had his stroke.

Grant

And what Paul had always learned was that certain parts of the brain are responsible for certain functions.  What he was told by the doctors is that the stroke caused memory loss, his father couldn’t speak, he was paralyzed, and they told him this condition was permanent.  That he should take him home and just expect him not to live much longer.

But Paul and his brother George, who also studied at the same medical school, they just couldn’t accept this.

Roberts

Dr. Paul Bach-y-Rita and his brother George nourished their father back to health.  despite what doctors told them about Pedro’s prospect for recovery, the brothers were able to teach their dad to feed himself again.  A huge accomplishment.  They taught him to talk again and to walk again.  Pedro even took up hiking.

Grant

But over time his father did regain all of those functions.  And he fully recovered.  So, they learned that the brain was so magnificent that it could recover from loss.

Roberts

What Dr. Grant describes is an example of brain plasticity.  Brain plasticity is foundational, the theory of sensory substitution.  The idea that a person can substitute one sense for another.  I asked Dr. Grant to give us an example.

Grant

If you remember when you’re younger there is that game where you’d draw a letter on your back – someone might draw a letter and say “guess what I’m drawing.”  And you’d identify the letter.  That is another form of sensory substitution.  Using tactile sensation to understand something visually.

Roberts

How is that the same or different from Braille?

Grant

Braille is a very basic form of sensory substitution.  But people who read Braille are using their fingers, so they’re using tactile input to experience a sense of vision.  It really is very similar, but when you’re looking at something like the BrainPort it’s obviously much higher technology and a lot different because it’s using the tongue which we’ll expand on a bit later.

Roberts

This may seem over simplistic, but remember, sensory substitution is not about the patient using a new technology.  It’s more about what’s happening at the level of the brain.  It’s about the patient developing an alternative sense.  BrainPort is a product Dr. Grant is working on that does just this.

So, if I was a potential user of BrainPort, how would you describe the BrainPort device to me and my family?

Grant

So, they’re going to wear a headset, and the headset has a camera on it.  The camera is going to act as how the eyes would.  It’s going to pick up visual information in their environment.  And it picks up that information in greyscale images.  With those images, the areas of the image that are white are going to produce a high stimulation on the tongue.  And the areas that are dark are not going to produce any stimulation at all.

Between that contrast, users are able to identify different objects and different features in their environment such as doorways, sidewalks, signs, obstacles.  Things like that that are very important to mobility and just orientation.  Understanding where one is relative to an object.  Relative to another person.  And then what else is in the environment.

Roberts

The BrainPort device is meant for people who are blind.  So, it’s not crowding out a person’s residual vision.  And surprisingly, both users who are congenitally blind, and users who have seen before and have a visually memory, have performed the same in clinical trials.  This shows that users are not experiencing a memory of sight.  They are truly learning to interpret the camera’s image through stimulating the nerve fibers in the tongue.

Grant

I was training someone.  We were walking the hall and I had a laptop in one hand.  I had my hand on my hip and I was waiting for them to respond and they looked over at me and they mimicked exactly how I was standing.  I was like, that’s really fascinating.

Roberts

So, tell me about that one patient.  About the patient that you remember most.

Grant

I think it’s the first patient that we trained for the clinical trial.  He’s an older gentleman who was in his 80s.  He was a socialite.  Very active.  Just such a great person.  His character and the things he talked about.  It was just a very exciting person to talk to.  But he had lost so much of everything because of his vision loss.  And we put the BrainPort on him and started training him and we were doing some mobility tests.  I was wearing – that day I had on a darker shirt.  Kind of a black shirt, and we were in kind of a lighter grey environment.  What the BrainPort can do is invert the scene.  So when he did that, the shirt that I was wearing became white, and I just remember walking around the room, and I was being very quiet and I think even took my shoes off, because I didn’t want him to use any auditory signals.  I was walking around the room and he would just scan the room and then all of a sudden I could feel when he perceived me.  It was just the strangest feeling, but as soon as I felt that he could perceive me he would just point to me.  And it was really so exciting.  I do think there are some terrors there and it was really, I felt like he was a person that this was becoming almost like sight to him.

Roberts

So, you could feel what he could see?

Grant

Yeah.

Roberts

Explain.

Grant

It’s such a hard thing to explain.  It was a feeling, but…

Roberts

You might say she sensed it.

Grant

I would always know.  As soon as he scanned the room and he spotted me I would just know before he even told me.  It was just like a connection.  The capabilities are so much wider than we even know.  And I really do believe that.

Roberts

Do people really “see” with the BrainPort?  Can you describe what happens when stimuli are received on the tongue and what’s the user experience?

Grant

Sure.  So, people who use it, they always do use the word “see” and I really discourage them from doing that because it’s not the definition of vision that the world understands.

It’s really shape recognition, and then kind of using other cues in the environment to fill in what that might be.

Roberts

Training is required to use BrainPort.  And more extensive training to truly learn to use an alternate sense.  I asked Dr. Grant to describe some of the challenges this technology has faced.

Grant

Because it is something that uses the tongue, because it uses a theory of sensory substitution that’s still is more widely accepted, as we know, but it’s not fluid in clinical practice yet.  So there’s still a lot of convincing of the technology that needs to be done.  And that’s done through manuscripts, papers, clinical trials.  Wicab has been really fantastic about not just premarket clinical trials but post market clinical trials and continuing to provide evidence for its use.

So, if the biggest barrier has been acceptance in the practice and also not just acceptance, but also the cost of it, because these types of devices are not covered by private insurance, by Medicare, Medicaid.  So, that has been a barrier, too, that we’ve been really working with different consulting companies to try to push it forward and get it approved.

Roberts

So, as you look forward to future iterations of BrainPort, how can you see this technology evolving even further over time?

Grant

There are other companies doing other types of things like auditory feedback, so there are areas of collaboration, I feel.  Because the great thing about the BrainPort is that it gives a person their own sense.  It’s not as if anyone has to tell them, oh, that’s a ball there.  That’s a table there.  It’s something that they can experience on their own.  And that is of great value ot a person who is blind.

But there are other opportunities as well where it might be like, okay, that’s a piece of paper, and if you pick up the paper it reads the paper.  That’s where I hope to see the device go.  That we can continue to add those type of features.

Roberts

So, I love this idea of marrying two great technologies.  And we’ve talked in the past in our podcast series about converting images into sound.  And of listening.  And so, tell us how you could see BrainPort working in conjunction with something like OrCam.  We featured OrCam in our first episode, “Seeing With Your Ears.”  The OrCam device relies on a camera mounted on eyeglasses and uses audio to translate the visual world for the user.

Grant

That is a thought.  OrCam has really great auditory technology.  If OrCam could read a book, how could a person who’s blind know where the book is unless they’re feeling for it, or they have another – they need another perception.

Roberts

Wicab, the parent company that created BrainPort is embarking on clinical trials in France.  The French government has agreed to pay to put the device in the hands and mouths of users.  This is one step forward for Wicab since the device is still quite expensive.

But the research that’s come out of BrainPort, the science that has been uncovered has contributed a lot to the understanding of sensory substitution.

Where do you see sensory substitution going down the road?  What excites you when you think ten years out, for example?

Grant

There’s the technology called North, and it is a chip that is implanted on the chest and it vibrates every time the person faces in the north direction.  It’s kind of like you’re a human compass, but that you always have a perception of where you are in the world based on the feedback this device is giving.  And I just think that’s another fascinating component to this whole field.  Their interest is not only for people who have impaired sensory modalities, but it’s also for people who want additional senses to believe there’s more that we can do with our brains than what we know today.

Rizzo

What’s beautiful about it is that the brain is meta-modal.  I think over time we can really do a very nice job with sensory substitution by taking one sense and then plugging in kind of altered sensory input.  So, imagine I were to create soundscapes of an environment.  It’s not just like the sound on your left would mean an object on your left and something on your right is the sound on your right.  It’s not just spatialized sound.  There’s more to it.

Playing with frequency and tone and pitch, and if you train for long periods of time you’re able to really do much better and to have kind of a much richer experience than kind of recapitulate it through the other sense.  The problem is there are these massive bandwidth limitations and differences between vision and the other senses.  Vision is so complex.  And some people estimate even if you compare vision to touch, for example, there’s a 500 to one difference in bandwidth.

You have so much that you would have to try and describe and so the idea here is, we can do a lot of training and provide these different experiences, and we can substitute up to a certain point, but then at what point do you saturate?  And how much are people going to want to tolerate with training in order to get to that point?  Unless trained exquisitely, I think it can be a bit problematic.

But what’s beautiful about sensory substitution is, if you take these devices and use the kind of reweighting or kind of recalibrating that this allows you or affords you, and you tie this to your vestibular system, then you have this amazing experience.  So, if you hook this up to the vestibular system and work on balance training, the results have been dynamite.  Very strong findings.  I have to show that you can kind of use this additional sensory input to kind of recalibrate the balance system, if you will.

It’s a complicated subject matter.  I do agree with sensory substitution.  I think benefit from those principles and use them in a lot of our research initiatives.  Simply put, I just really want to amplify your existing senses and to augment what I can give to you right now so you can have a richer experience.

Roberts

Dr. Rizzo is developing a wearable device that he will launch under the commercial name Sensory Halo.  He calls the theory underlying Sensory Halo “sensory augmentation.”

How is sensory augmentation different from sensory substitution?

Rizzo

It’s more intuitive and you can do that without kind of laborious training intervals.

Roberts

Dr. Rizzo has a lot of questions to answer about the Halo before he brings it to market.

Rizzo

We have a lot of fundamental research to do first, which is very important to make sure this device really performs well, computationally, power wise, where are the constraints in the system, what are we going to need to do?  How can we leverage existing systems and other platforms to advance where we’re at and accelerate where we want to go?  And so, what I mean by that is, if I have a wearable, can I do things like cloud computing or using public Wi-Fi?  Can I do things with better battery technology or energy harvesting from the actual device itself through the vibratory interfaces?  Are there different things I can do to make it run faster for you to actually work with someone else who has a similar device to do a pact or a multi-agent approach.  All things along those lines.

Roberts

Where is the Sensory Halo in the development process?

Rizzo

The device itself is definitely being developed.  It has both an academic presence where we’re advancing the fundamental aspects of the research.  And the good fortune of being issued a patent by the USPTO in partnership with NYU.  So, NYU holds the rights to this patent and we’ve spun out a small startup.  We hope to eventually launch this commercially under the name Sensory Halo, providing a halo of protection for you.

Roberts

Dr. Rizzo’s device is designed to empower the wearer by delivering key pieces of information about a navigational environment.  We will link to a presentation he gave about it in the show notes.  His passion for this project comes out of his own experiences navigating with vision impairment.  His primary goal is safety.

Rizzo

Very seldomly do we have a human companion that can chaperone us everywhere we want.  We don’t always want that, right?  I think there’s this kind of notion in some areas that disability is weakness and that’s nonsense.  In my book, disability is strength.  There are so many incredible individuals with disability that have made them just incredibly strong.  we just want to create some power tools for you to be able to successfully navigate through life.  Though we whole-heartedly know that you are unfortunately put in a lot of dangerous situations.  The idea is how can we better protect you by translating something analogous to that human chaperone, audio cue and hand code into something that would be very streamlined.

And where were at right now in this fourth industrial revolution we wholeheartedly believe that creating these really neat, new instrumented book bags or even these vests that have these very inconspicuous camera systems you don’t even notice can provide a lot of real time mapping for you.  And then give you dynamite views to help keep you safe.

Roberts

We asked him what excites him most about the fourth industrial revolution.  And his answer is not what you might expect.  It was cars.

Rizzo

The majority of these driverless cars depend on what are called advanced driver assist systems that are helpful to do everything from lane keeping, animal detection late at night to emergency brake applications.  What’s been very dramatic is if you look at the massive reductions in collisions and how many lives that’s saving.

Roberts

Dr. Grant suggested something similar when we asked what excites her about the future.

Grant

If cars can drive themselves, why can’t we put people with the same technology to navigate the world more independently, more freely.

Roberts

And the OrCam device that Dr. Grant mentioned was actually created by the founders of Mobileye, an advanced driver assist technology.

Rizzo

You can go on Amazon and buy a kit of ultrasound sensors for a bumper for like $25 right now.  And it’s amazing what you can do with kind of building part of this augmented wearable by using these little ultrasound sensors.

Roberts

Of course Dr. Rizzo has used some of these ultrasound sensors in his device.

Rizzo

We’ve used all kind of ultrasound sensors.  Just kind of LiDAR sensors and we’ve used IR sensors, and all different types of thermal imaging sensors.  We’ve used many different types of what are classically called distance and ranging sensors in addition to optical sensors for the wearable.  It’s really been a ton of what I call innovation surfing off of the wave of development from driverless car.

Roberts

You can hear Dr. Rizzo’s excitement in his voice.  As a physician-scientist, how did he become so passionate about product design?

Rizzo

Sometimes people will refer to rehabilitation doctors as the aluminum kings and queens of medicine when you really think about how to use crutches or how to dispense a wheelchair appropriately or what’s involved in power chairs.  A lot of the new designs you’re seeing are a by-product of this kind of amazing marriage between both rehabilitation and engineering and also superb clinical acumen.  And that’s damn cool.

Roberts

The ophthalmologists agree.

In this episode we explored the big ideas of brain plasticity, sensory substitution, and sensory augmentation.  One of my big takeaways from this conversation is the long game of science.  Just like new technologies surf the wave of development, as Dr. Rizzo said, new scientific knowledge is built on research and discoveries that came before.  This is why the findings around brain plasticity and sensory substitution are so exciting.  Precisely because there is so much more to be uncovered.

Did this episode spark ideas for you?  Let us know at podcasts@lighthouseguild.org.  And if you like this episode, please subscribe, rate and review us on Apple Podcasts, or wherever you get your podcasts.  I’m Dr. Cal Roberts.  On Tech & Vision is produced by Lighthouse Guild.  For more information visit www.lighthouseguild.org.

On Tech & Vision with Dr. Cal Roberts is produced at Lighthouse Guild by my colleagues Jaine Schmidt and Annemarie O’Hearn.  My thanks to Podfly for their production support.