On Tech & Vision Podcast
Extending Human Vision
On Tech & Vision Podcast with Dr. Cal Roberts
Today’s big idea is how the technology used in instruments that extend human vision to space is being relied on by vision technology developers in devices that help people with vision loss in everyday tasks here on Earth. Using substitute senses has allowed scientists across many fields to continue their work without the use of sight. The eSight is one such device that stimulates the remaining functioning vision to improve the quality of life for users. Dr. Roberts speaks with Charles Lim about the development of the device, the principles behind how it works, and the motivation for future improvements.
Podcast Transcription
Lambert
Almost like our Milky Way, it actually spirals out with almost three legs, that spiral out it. It’s almost like you want to grab it and almost spin it.
Roberts
Denna Lambert is a project manager at NASA’s Goddard Space Flight Center, who was born with congenital cataracts. She’s using her fingers to describe the shape of the spiral galaxy NGC4603.
Lambert
That’s actually what the galaxy is doing. It’s spinning.
Roberts
This clip is from a video that we’re sharing with NASA’s permission in which Miss Lambert introduces viewers to Touch the Universe: A NASA Braille Book of Astronomy by Noreen Grice.
Lambert
It’s kind of cool. It either feels almost like a spider with legs, but you know that it’s a galaxy in space.
Roberts
Touch the Universe uses textures and shapes to recreate Hubble Space Telescope’s most famous images. Nebulas, planets and galaxies converted into a tactile sensory experience. A Braille experience of the universe. This is one way that people who are blind can access the wonders captured by the Hubble Telescope.
But there are other ways these images can be made accessible. That sound is the sonification of a Hubble image. That’s right! The data in the visual image, in this case an image of many bodies in space, has been converted into sound.
This “Astro Music” was made by Matt Russo. But sonification, the conversion of astronomical data about stellar events and black holes into sound, is a tool for understanding space that has been championed by Dr. Wanda Diaz-Merced, an astrophysicist who lost her sight due to diabetes when she was a physics student at University.
Diaz-Merced
At the back of my head my thought was, “What am I going to do in physics?” Everything is so experimental. I cannot do the drawings. I cannot do the experiments. I cannot do the theory. What am I going to do in physics?
Roberts
Her friend shared with Dr. Diaz-Merced a system he designed to reproduce data as audio.
Diaz-Merced
When he reproduced the audio of the data, I said, well, perhaps there is an option. Perhaps I can convert numbers into sound, and I can listen to changes into the data and do the physics. In this case it was astronomy.
Roberts
By listening to the stars, Dr. Diaz-Merced was able to uncover something new about supernovas and gamma ray bursts.
Diaz-Merced
When the star reaches the supernova stage it releases energy, right? What we did is that we converted into sound. Data from a gamma ray burst, we were able to listen to small variations in the data that were not visible to the human eye. What I found was those changes that I had heard were indicative of resonances, and resonances are exchanges of energy between particles and waves. In resonances that were happening at that moment were indicative of elements that are also very important.
Roberts
The elements that Dr. Diaz-Merced found evidence of in these resonances pointed to the birth of a new star. Dr. Diaz-Merced’s discovery proved that astronomers and other scientists who are blind can make meaningful contributions to their fields by using substitute senses, especially in fields where instrumentation does most of the heavy lifting.
Beck-Winchatz
Every human being is basically blind in almost all available light.
Roberts
Dr. Bernard Beck-Winchatz is an astrophysicist at DePaul University. He worked on the Hubble Telescope and with Noreen Grice on the book Touch the Universe and has insights on the limits of sightedness when it comes to astronomy.
Beck-Winchatz
There’s a wide range of light waves or electromagnetic rays as they are more technically called that range from gamma rays, x-rays, of course we’re familiar with, and ultraviolet rays. There’s a very small range of what we call visible light that sighted people can see. And then on the other side there’s infrared light, and then there’s radio waves, micro waves. All of those carry information about our surroundings. They’re all important, not just light that we call visible.
Astronomers have realized that early on you can learn a lot about the universe by developing instruments that can be extensions of our own senses.
Roberts
Just like astrophysicists rely on the instrumentation inside the Hubble Telescope to extend human vision into space, 340 miles above earth, in fact, vision technologists also rely on instruments to be extensions of our senses. That’s today’s big idea.
This is On Tech & Vision with Dr. Cal Roberts.
I’m Dr. Cal Roberts and our guest is a technologist who has dedicated his career to using cameras for vision technology. Charles Lim is the Chief Technology Officer at eSight Eyewear. In his early career as an electrical engineer he actually worked on the Hubble Telescope for a company called NDA Space Missions. He then moved to IMAX which also relies on cameras to be the eyes, so to speak, for viewers in the cinema.
How do you go from being a young electrical engineer to someone passionate about instrumentation to improve people’s vision?
Lim
When I first graduated I actually joined the NDA Space Missions and we were essentially working for them to actually build the vision technologies that would go into the Hubble Telescope’s spacecraft mission, which also was similar technology that we have used for some of the shuttle Discovery, for example.
That was my first at understanding the hardware vision capabilities, the softer vision, which was relatively high tech back then.
Roberts
What kind of work did you do helping the Hubble Space Telescope to see?
Lim
As an electrical and computer engineer, I dealt with all sorts of processors, lenses and image sensors as well. And what that meant for me when I was starting out in my career was, how do I combine that to actually allow for a robotic vision to happen in space? It was relatively exciting because we needed to make sure that the sensors were accurate enough that the machine itself could understand what it was looking for.
We needed to make sure that the image itself could be segmented into smaller pieces. You can think of it as, if you had a TV, we would convert the TV into 64 small pieces and how do we know which piece to look at next to make sure that the robot itself understands what it’s doing and where it’s positioned to?
The other aspect is, because we were in space, unfortunately radiation is a problem. I literally had to go to TRIUMF Cyclotron in British Columbia, Vancouver and wear unique radiation protection gear to actually make sure that the actual device could withstand the radiation out there in space.
We were testing it from minus 40 degrees to extreme degrees in temperatures to make sure it was relatively solid.
Roberts
And then, from Hubble you moved to IMAX. Tell me a little more about how your work at IMAX relied on vision technology.
Lim
IMAX is a very immersive type of cinematic entertainment company. We nit-picked everything on the vision side. As you can imagine, if you blow up any type of image on a large screen, what happens is all of the imperfections also get blown up. And having to deal with high profile directors who could literally amaze me by seeing one small broken pixel in like a big screen, we needed to make sure that everything was relatively perfect.
Roberts
What was the nitty-gritty like?
Lim
It was all about contrast, brightness, resolution, images on the big screen, analyzing the implications of 2D images, 3D images. How do we make sure that the left image is not too far from the right image? And when they actually watch that 3D image that it provides them the best 3D image possible in the market.
So, I’ve been involved in vision in one way or another, whether it’s in the capture portion of image sensors, or to the playback portion of IMAX Corp like cinemas. Or to the distribution and delivery. But what it all means is, how do we leverage the technology advances in cameras, image sensors and processing to allow machines and now, more importantly, our users and humans, to enhance their vision through more information?
Roberts
Let’s go back to the beginning with eSight. Tell us about the founder of eSight, Conrad Lewis.
Lim
Conrad Lewis was our founder, and he is a brilliant, successful electrical engineer. He had two sisters that were born with Stargardt disease. His main objective was, how do I actually help my two sisters see better? That has been the start of the foundation of eSight. Over the years he’s been gathering and getting a lot of experience with gadgets, software and the end result of that is he wanted to create a product that could not only help his sisters, but also could help other people living with visual impairments to see and live more independent lives while being mobile. And that basically became the mission and the foundation of eSight ever since then.
Roberts
So he built this early prototype. What did it look like? What did it do?
Lim
Well, as with all good things in terms of new prototypes, you can imagine it was relatively bulky. We actually had a tech wire connecting the headset to the controller. The actual controller itself, you can think of it almost the size of an X-Box game console. The weight was still relatively heavy on the head. And let’s say the ventilation wasn’t quite there and the software was a little bit simplistic. This was after we invested seven years and millions of dollars in R&D development.
But, I think what it has done for us is proven out the technology and allowed us to realize that yes, with technology we can improve the lives of our lower vision users. And clients, even though, of course, the first prototype was a little bit, let’s say bulkier and heavier and less usable than we’d like, but the concept was definitely proven.
Roberts
Explain to me the eSight theory. The device. How it works. Who’s it designed to help?
Lim
The eSight design itself is designed to help people with low vision. A good example could be Stargardt disease, macular degeneration, diabetic retinopathy. Most people have conditions where portions of their eye are damaged or not working. And the surrounding cells around the eye also tend to be dormant. Because it’s closer to the damaged portion of the eye.
But what we found is that, if we provide the right stimuli, whether it’s contrast, brightness, movement, temporal, what happens is that it allows us to leverage the dormant portions that are still generally working. You can think of it as, if you have two small TVs that are specifically catered to provide that much more information for you, whether it’s higher brightness, higher contrast or better acuity with magnification. This allows you to essentially see better and regain a lot of the previous functions and activities that you otherwise were not able to do.
Roberts
I’ve seen pictures of this device. It looks like a pair of special glasses.
Lim
I would say it looks like a special pair of larger glasses. So, if you wanted to take a look at it, think of it as like a halo type of device with glasses version type of HMD in front.
Roberts
It reminds me of Geordi LaForge’s headset from Star Trek: The Next Generation.
Lim
Closer to something like, if you were to look at maybe HoloLens from Microsoft. That would be a good example. VR headsets tend to be a bit larger so it’s actually smaller than that, and lighter. We made it so that it’s relatively comfortable. Relatively sleek looking. While also making sure that you have the wireless capabilities, the detachable battery so you can use it for a lot longer. We took this all into account as we were creating the form factor for the design.
Roberts
So someone wears this, it’s a halo you said. It’s a band that goes around their head rather than just a pair of glasses that hooks behind their ears.
Lim
Exactly. What we realized when we were talking to a lot of our users and clients is that it’s not necessarily the weight of the device that is a problem. It’s more of it’s heavy on the front. That means for us if we want it to be as comfortable as possible for our users, we wanted them to be as balanced as possible so that when they wear it, they don’t actually feel it. Which means that the halo band actually provides us the best combination in terms of flexibility, how to wear it and to adjust it to balance the weight on your actual head when you’re using the device.
Roberts
Give me an example. So, when someone puts on the eSight, what do they see?
Lim
We’ve had users, for the first time they said, oh, I can actually read my appliance manual now because I could never see it before. We have also had some of our users been able to go back to school. Some of them have been able to use eSight to go back to work. We have been able to get people with relatively bad eyesight from 20/600 all the way back to 20/20 vision which, in my view, is amazing.
Roberts
When I think of different types of devices I think of certain devices that work best if the viewer is not moving. And what they’re looking at is not moving. For example, someone sitting on their sofa and looking at the television. Then there are devices that work very well if the person is not moving, but what they’re looking at is moving, If you’re sitting in the car and the other cars are moving but you’re not moving. And the third type is situations where you’re moving and what you’re looking at is not moving. so you’re walking down the street and you’re looking at a sign. Was eSight developed for one or more of those markets?
Lim
Good question. eSight was designed so that we can essentially allow all of our users to live their normal lives as much as possible. And if I had to categorize into any of the three buckets, I would say probably more of the other two even though we can still do the first one. Let me explain that a little bit.
Nothing stops us from getting our users to sit down and watch TV or even use their computers. In fact, we had made sure that we have the ability to plug in HDMI cable inputs so that they can actually directly see it onto their display. Nothing stops us from that, but I think the really unique part of eSight is its mobility. which means you have the mobility to walk outside with our device and be able to see the surroundings around you.
You can sit in a car and also enjoy all of the other items moving around you. And, we’ve made it easy for our users to actually move the device with biopic tilt if they wanted to use their own vision, and we are not blocking their peripheral vision, similar to other VR devices where, if you put it on, you’ve pretty much lost your whole peripheral vision.
That allows our users the best of both worlds in terms of trying to live their normal life as much as possible because they have their mobility, their peripheral vision, but it doesn’t stop them. We can also leverage a lot of the information coming through to them on their phones, whether it comes from their mobile apps or the mobile phone sensor, such as GPS, or accelerometers, and make sure that we present that in a way that our users are able to understand it better.
A good example of that would be our device actually allows up to tap into the display of the mobile phone they have and see it directly onto their HMD device. Because we are now connected to the cloud, we have the ability to actually share that both ways, which means they can connect to their caregiver or loved ones, which might be half way around the world and share with them.
Roberts
Are you finding that your users are wearing the eSight most every day or maybe a couple times a week, or a couple hours a week? It’s hard to say what’s typical, but if you just kind of give me an average so I can understand better to the extent that the eSight is becoming part of people’s lives.
Lim
Previously, all devices ran out of battery after one or two hours. But the fact that our users are asking how can we use it for longer has shown to us that they actually want to use it as much as possible without having to be limited by the battery capabilities. And that’s why we did the detachable batteries.
When we were talking to some of the people in the field, they said that this is great because it allows me to work a full eight hour day whereas before after using it for two or three hours I would need to recharge it. But, with the batteries that we have right now, I can take it to work and make sure it allows me to use the eSight device for the whole day.
Roberts
In the development of great technology, it’s never done in a straight line. You start this and it doesn’t work and you go back and you try this and you try that. And there are obstacles along the way. Explain to our listeners, what are some of the obstacles and how did you get around them?
Lim
It wasn’t easy, and certainly was quite an exciting last few years for us, but maybe I can share some of the more complex challenges that we had to deal with. Heat was always a problem. Because we wanted to make sure that the device itself was capable of living up to the expectations of our users which meant high processing power, battery usage and power drop.
We wanted to make sure heat is not an issue, so the new design that we have has done extensive modeling on the heat parameters of the actual device to make sure that it actually doesn’t have as much of a heat problem for our users, which mean new ventilation designs, heat syncs, and patented pipes that actually allow us to spread the heat all over the place and reduce them. That’s one.
The other one was the comfort. We have iterated the form factor multiple times. We have said, oh, this might work, and we’ve gone through four or five mechanical form factor designs. Because, whenever we would try it, either it’s still front heavy, or it’s back heavy or it doesn’t fit the head sizes of kids. And we wanted to make sure to cater it as much as possible. So, how do we make it comfortable and accessible enough that it would be comfortable for smaller people of our society all the way to the more senior people of our society.
Battery life was another one. How do we make sure that we provide the ability for all of our users to use it for the full day and not have to charge it for two or three hours while making it detachable? There was discussions about attaching a big battery on the back, but if you weighed that then all of a sudden it’s like you’re carrying a big brick on your head which is not good either. So, we had to balance battery density, power usage to make it optimized and make sure that we do hot swappable batteries.
The other interesting one was usability, for example. How do we make sure that we can make the device usable for the younger generation who loves technology, is very technology savvy, but also cater to our older members of society that some of them may not even carry a smart phone. How do we get rid of that and move over to Bluetooth and WiFi while managing EMI emissions, safety, making sure that the data rates are fast enough to make sure that the latency is not a problem?
More importantly, also making sure it is future proof. So, eSight 4 is upgradable in terms of its software capabilities. And we intend to make sure we roll out new updates to the device.
Roberts
So, is there an opportunity for this technology among sighted people?
Lim
There is. And I’m a big believer of this. We are fortunately living in an age where technology is being integrated of all aspects of our lives. We already are all heavily dependent on technology. Most people, and myself included, cannot live for much long without my smart phone, and being attached to it checking weather, email, Facebook. You name it, I use it all the time.
What that essentially means for us is that eSight, our goal is to make it an extension of all of our users, whether they’re sighted or low vision. Which means we want to improve the vision and the quality of life of everyone, starting with our low vision users. So, the next time, let’s say we have an older member of our community, and they might have their normal vision. But maybe they have a little bit less of a good memory. We can help them mention that, oh, you’re looking at your daughter, to help them refresh their memory.
Or we can even say that, for example, you need to take your medicine for today. So we can take inputs from all of their doctors and then give them some advice. Surgery is another example where we have a lot of interest from our professionals in the healthcare space to actually use our device to help enhance their capabilities in that space.
Roberts
So, what’s next? How do you see this technology developing? What challenges are you still working on solving?
Lim
Very good question. We are very plugged in to the latest developments in artificial intelligence and big data. And trying to figure out the correlation of all of those things with the eye conditions of all of our users to figure out what’s the best way to optimize it for them.
And I dream of a future where eSight can really become a natural extension of our user’s vision.
Roberts
You told me when started that you have experience both in engineering and business development. And so, the two sides of innovation. How do you think personally that that combination of experience and skills in getting vision technology in the hands of people – how do you think that’s really benefited you the most?
Lim
I’m actually relatively fortunate to have seen both sides of the equation. And I believe the combination of business development and technology allows me and my team to develop and bring better innovations to the market.
I have seen a lot of brilliant technologies out there being developed and forced into the market just because it looks like it’s a cool technology. Which is fairly engineering-driven. This results in products that are out in the market but not really solving any real customer issue or need.
The business side allows me to understand the main drivers for our sales, our partners, and more especially so, our users, so that when our technology team develops our products we make sure that it’s specifically catering and addressing a need in the market. This allows us to build the product with our community to ensure that they know that their needs are being heard which is very important in the development of eSight.
But, some of the aspects of the improvements we’ve done with eSight for are not just vision-related, with the wireless design, the ease of use, the balance comfort. All of these things have come from the fact of being out there in the field talking with our partners and our users and understanding what the community really would like to see in the next device so that we make sure that we take that into account. And I think that you really need to have both sides of the coin to be able to develop great products that are in tune with the market and not necessarily primarily catering to the next technology trend per se.
Roberts
Tell me about what this all means to you personally. Being able to see this technology grow and bloom and see how it helps people’s lives. How has it impacted you personally?
Lim
The ability to positively change individual lives as well as help create a more inclusive world is one of the best feelings out there. The ability to share your experiences with your own children, helping them to see humanity and abilities in a new light and possibly to aspire them to have their own purpose-driven impact is very satisfying.
I’ll give you a few solid reasons. For example, in my previous careers I have won multiple awards including Award of Excellence from NASA’s Goddard Space Flight Station themselves. But only in the last year when I was at a conference with eSight and Dell did I have parents literally hugging me, shaking my hand. And some of them even cried because it was truly a game changer for the child that now they can go back to school and study.
That is amazing. So I am very fortunate and it’s truly a privilege that I’m able to use my skills in a way that I’m helping people to live their fullest potential and for that I couldn’t be happier.
Roberts
And that keeps you motivated to do more. That’s great.
Beck-Winchatz
Our pupil has a diameter of half a centimeter, and the Hubble has a diameter of 2.5 meters. The area that detects the light is proportional to the square of that, so many millions of times more light gets into the Hubble than gets into your eye. And then the other thing is we don’t know how long the retina integrates the light that comes in, but it has to be just a small fraction of the second because you also don’t want to lose the smoothness of motion. So that’s the other thing. So those are the two ways which telescopes allow you to see things that are much fainter than what our eyes can see.
Roberts
We described the eSight concept to Dr. Beck-Winchatz and he had this suggestion.
Beck-Winchatz
The technology is making what we call visible light accessible to people with visual impairments. And you could imagine having a similar set of glasses that make radio waves visible, right? I think that’s kind of an example of the fact that what we happen to be able to see is just such a small portion. And we could all use these glasses. We’d all learn a lot more about the world we live in if we had these types of glasses in other wave lengths.
Roberts
Talk about big ideas! I hope someone listening gets on it.
At Lighthouse Guild we love big ideas. And we also believe strongly in accessibility for people who are blind or have vision impairment.
Diaz-Merced
They want to motivate students to follow STEM careers. But give them the tools to follow STEM careers. Give them the tools so they can perform at their own maximum and with the perception modalities in which they perform at their own maximum. But also based on our human right to equality we can no longer be deprived from equal participation.
Roberts
This is as true in space as it is on land. Just like the tactile book Touch the Universe makes Hubble’s images accessible to Denna Lambert.
Lambert
It kind of made me feel like a kid again to be able to say, Oh this is what a star looks like, or this is what a telescope looks like. Or this is what our planets look like. Because, for such a long time, I didn’t know.
Roberts
And just like Dr. Diaz-Merced’s sonification system makes astronomical data accessible to her and to other astronomers who are blind…
Diaz-Merced
We evidence that you use sound it increases to your sensitivity to the data by nature will be blind to the human eye. It means that people with other perception modalities can bring a lot to the science.
Roberts
So too, eSight makes the terrestrial world accessible to people with low vision.
Did this episode spark ideas for you? Let us know at podcasts@lighthouseguild.org. And if you liked this episode, please subscribe, rate and review us on Apple Podcasts or wherever you get your podcasts. I’m Dr. Cal Roberts. On Tech & Vision is produced by Lighthouse Guild. For more information visit www.lighthouseguild.org.
On Tech & Vision with Dr. Cal Roberts is produced at Lighthouse Guild by my colleagues, Cathleen Wirts, Jaine Schmidt and Annemarie O’Hearn. My thanks to Podfly for their production support.