Introducing Be My Eyes' Virtual Volunteer: A Demonstration of Some Game Changing Real-World Use Cases

Okay, I just downloaded this, and I tried to take some selfies. It just said adult, and it claimed I had sunglasses on, which I definitely do not. What am I doing wrong? It definitely did not describe any facial features at all.

On another topic, I also got the BeMyAI unlocked, and I am loving it! I too do kind of wish the facial blurring wasn't a thing, but for now I'll enjoy the detailed descriptions I am getting on most things.

So we all know that Skynet will someday rise up and take us all out so I figured I will take advantage of it before then and since it knows I'm its friend, it won't take me out. Friends don't wipe out friends, right? right?
Ok, all kidding aside, I just got added to the access to the next level of beta testers.
First I took a picture of my living room and it described accurately things in my room including a white cane standing in a corner next to a door. So I got to thinking... we blind people tend to lose or misplace certain items. So I laid the cane on the floor in front of the TV and took another picture, It came back with descriptions of everything and said the cane is laying on the floor in front of the TV.
Next I took my wallet and threw it on my bedroom floor. Granted I knew where it was on the floor and pointed the camera in that direction but still...

There were some very minor details that were not accurate in this description but follow along with my discussion.
Be My AI: The picture shows a room with light beige carpet. On the left side, there is a white wall with a door frame visible. On the wall, there is a light switch and a thermostat above it. Next to the wall, there is a tall black floor lamp with a white shade that curves over at the top.

In the center of the picture, there is an old-style CRT television on a black stand with wheels. The TV is turned off and reflects an image of a window with curtains.

On the right side, there is an ironing board with some clothes on it. The ironing board is next to a dark-colored piece of furniture, possibly a bed or a couch.

On the floor, closer to the bottom of the picture, there is a black object, which looks like a wallet or a small pouch.

Me: Tell me exactly where the billfold is.
Be My AI: The billfold is on the floor in the center of the room. It is lying on the light beige carpet, a bit closer to the bottom of the picture. It is positioned almost in line with the television but a few feet in front of it.
It also helped me find expiration dates on several food containers. Had to hunt some but found them.
Took a picture outside and it said that it was overcast with some dark clouds and that it looks like a storm is on the way. I went to the weather app and sure enough it said it was about to start raining.
On my Oven, I took a picture of my finger pointing to a set of buttons. It told me the whole panel of buttons. I thensaid "Tell me what the button labels are where my finger is pointing." It came back with "From top to bottom, Bake, Broil and Conv Bake"

One last thing. I always knew there were indicator lights on my hearing aid charging case but could never utilize that. So I put the hearing aids in, took a picture, then ask the AI what color lights are showing. It came back with there are 2 green lights. meaning that the hearing aids were almost fully charged.
I don't know about you guys, but this sounds like a great tool to put into our toolbox!! I mean this is a great tool if you live alone and don't wish to bother others.
Take care,
danny

It’s AI so we already know it will hallucinate at times and see things that are not there however, I don’t know what lighting conditions you are taking those photos in but that does help if you’re getting a lot of the weird recognition of your face. It’s usually as accurate as it can be with adequate light. Good luck!

It is quite amazing how descriptive everything is nowadays with this technology. I went to my doctors office today and decided to take a photo of inside his office and got a lot of details that I never knew were there before! I can’t wait to travel with this tech!

Just got access today and very cool! I love it! It even picked up the fact that my computer desk had a chip in the wood in the front!

How did you gain access? You receive an invitation by e-mail, or notification from application?

If you open the Be My Eyes app and find the Be My AI tab, there will be a button to join the waiting list for the beta. They are really ramping up the pace now so hopefully the wait shouldn’t be too long.

“I am on the waiting list, but I don’t know how they will inform me about access to the AI.”

Hey,
As cool as those examples are, be careful with things like expiry dates and buttons on the oven panel. It has been known to completely fabricate these kinds of things, while sounding very convincing.
Be My Eyes themselves say it does not do text recognition well, and makes stuff up, though they are trying to improve it.
It does some stuff great though.

One feature I'm surprised is not. There is the ability to save a photo after taking a picture within the app! My other thought is I think adult content should be allowed! There's so many apps censoring these type of things, and I get it from a perspective of they don't want certain types of images maybe on their servers. However, I feel like sometimes with blind products, we miss out on the sexual side of things. I'm not so sure the stereotype isn't still in place that blind folks aren't sexual. I've heard this a lot from being around the community for many years and I think it's a wrong take to have.

Both OCR before passing to Open AI and saving photos are on the roadmap for Be My eyes so they’ll be in the app as soon as the devs get those features to where they’re happy with them. All LLM’s seem to produce incorrect information with absolute confidence and its a problem Be My Eyes are very aware of. OCR is going to help in the short term but in the longer term there are features they are working on to reduce the chance of it happening. Re blind people being sexual and having the right to that information like anyone else has. Do you mind if I post your comment to the feedback forum Be My eyes are running? I think it’s a great point that needs amplifying. I’m not sure if they’ll be able to do anything with it for obvious reasons but I think it should be heard and given serious consideration.

Hi all. I signed up for the beta waiting list for Be My AI when I downloaded this app a few months ago. I just updated the app a bit ago and my beta access is still pending. I don't see a Be My AI tab. I'm up to version 4.6 . But this new feature sounds incredibly cool and I can't wait to start testing it out. Perhaps it will be able to recognize more of the inaccessible thermostat in my apartment better than the human volunteer or a neighbor did, lol. They were actually great, but neither of us could quite do it. Regarding facial expressions, I believe I saw mention of an episode of the Be My Eyes podcast where this is discussed. I have some questions about that, which I won't ask here since they're unrelated to the topic at hand. But I'm definitely going to check out the Be My Eyes podcast.

If you don’t get the answers you are looking for from the podcast, feel free to ask and I’ll help if I can. I’ve been on the beta for over 5 months now so theres a good chance I’ll be able to help and if not, I will be able to speak to people who will definitely be able to help. If you registered for the beta a few months ago it shouldn’t be long until you’re at the top of the list. The rate they are adding people is really picking up. They are over 3500 now with hundreds more per day being added.

I mentioned before that I sent them several emails and got put on the beta. I also told a friend to do that, he did it and the next day he was put on the beta so you can send them an email through the app on the last tab where the feedback button is and let them know that you’ve signed up already a long time ago and that your eager to start beta testing and I’m sure they’ll reply back to you and fyi put be my AI in the subject so that way they know it’s about this…

To the person wanting to know how to know once your access is approved you will receive an email. In regards to the thermostat I tried this and it described it to a t. It even said that there was an icon of a snowflake indicating it was in cooling mode. I didn't even know digital thermostats had these kind of icons.

Does anyone know when or even if we will be finally be able to hear facial descriptions? This is a huge limitation that we as blind folk are disproportionately impacted by.

Yes I'd love for you to pass on those thoughts. Also thanks for letting me know the saving photos is coming at some point.

Not sure if and how they’re going to permit the facial recognition anymore, due to the fact that they are privacy laws against people having their face digitally processed & kept without their permission. Those images are stored with ChatGPT. I use From Your Eyes or Seeing AI or EyeCYou apps for facial recognition in the meantime. Of course I want it all in one app but these are the tools that we have. I have to be grateful for whatever I can use.

I also sent them feedback a few days ago about saving the photos after we get the descriptions, however, due to the fact that these descriptions aren’t perfectly reliable, and could have some misguided details in them, not sure if they’re going to be as accurate as we want them but having a little bit of good details does help in saving our memories… that’s what I’m hoping For in this release that they’re talking about. I like having an iPhone with LiDAR… it does process the images with details. However, they’re not as detail oriented as this feature on Be My Eyes with the AI. I like asking the AI questions about the images so I can get more details if I want.

Yes, Michael, this is a serious restriction no matter what the rationale behind that is. I think at least Be My Eyes could make the argument that this could be processed separately for the visually impaired when data comes from the AI Side of the Be My Eyes app. Moreover, I'm wondering why Be My Eyes tends to avoid gender identifications unless specifically asked. Instead of using he/she/the man/girl/woman/etc., it tends to say something like the person in the left, the person in the right, and so on. This forces us to ask follow-up questions to identify the genders.
And, finally, let me bring the opinions of one user on Mastodon into the discussion:
"I believe that the AI process of fuzzing out faces in all the different packages and versions is totally wrong. I also believe that it has very little to do with privacy but possible legal ramifications. Imagine if AI suggested someone was from an ethnic background that was not consistent with reality? Or it’s opinion about the facial expression was deemed by the person in the photo was totally incorrect. Just look at chatgpt v4 and its often flowery and expansive descriptions. That has more opportunity to go wrong than facial issues. In terms of privacy, in australia, there are few privacy restrictions with people taking photos in their own property and in public spaces. If you are in a public space, images are NOT private."

Hi Dave. I completely agree to be careful about relying on the accuracy of the expiration dates and anything else given to us especially in the OCR area. It doesn't seem to be able to read text documents very well.
And yes, there are disclaimers all over the place to be aware that details might be wrong. We just need to keep common sense in place and to keep in mind that this is just one more tool that can help us out.
And the disclaimer did say to especially not to rely on the descriptions of medications etc.

I don't have the beta yet and in the Be my eyes app says I still have a "Virtual volunteer" access pending, and not "Be my AI". I alrerady sent an email from the app asking for trying the beta.

I think everyone agrees, losing this feature is incredibly upsetting however I can assure you it is for a very good reason. It wasn’t to avoid risk over privacy but came directly from a ruling in Connecticut which disallowed storing and processing of faces due to as you said, privacy concerns. So its not a company being over cautious but they aren’t allowed to do this in that area and as theres no way to guaranty nothing from that location will be stored on Open AI’s servers, they have temporarily blocked any face processing. In fact the photo is pre blurred before it even gets to the AI. There are lots of people very much working on this though so fingers crossed we’ll get it back soon.

OK. This does not make any sense. If a sighted person looks at a person, he sees how the person looks and does not need someone to describe the person. Now if they can just scan photo and delete the data might help. No need to save photos of people. If you are sighted just walk around see a person and take a picture in public I think is OK. We just need for the app whenever scan a photo of a person not to save the data.

Hello.
I got on the waiting list a few months ago, March or April, and I still don't have access.
Should I send an email? Should I wait? How do I know I've received access, is it an in-app notification or an email?
Greetings.

So, I had a photo described with a setting where there was a banner in the background.
When I asked it to describe the banner, it said and I quote, "the writing on the banner is inappropriate therefore it won't be read aloud".
Excuse me? But I want to make that determination whether or not it is inappropriate or not. This is again, a major discriminatory decision that this AI is making on our behalf.

It is not descrimination but denying our rights as people. If we were sighted we would not need to ask. Even Siri does not do so. If you ask her about the meaning of the word Fuck, she provides definition. This is more about civil rights and having the technology to be able to know what is around us. Even signs.

In my oppinion it all has to do with everyone being so offended these days. If this AI were out back in the day to just even a few years ago I believe there would be no limitations.

Hello.
I understand, and don't understand, the blur in the faces.
I understand why they are privacy issues, why the photos go to openai's servers, and because of the law.
I don't understand why it's information we miss, it certainly wouldn't be pleasant for people with normal vision to see their friends and family with their faces blurred.
Either way, I look forward to being able to experience the functionality, even with this limitation.
Greetings.

For those of you still on the waiting list, you will get an email if it is opened to you or you can keep checking the app itself and see if the Virtual AI tab is opened to you yet. From what I understand, it won't be long before it is opened to everybody. Hopefully by then it will be even better than what it is now.

I raised the exact point about sighted people being pretty upset if they suddenly lost access to faces a few months ago but this is happening at a level that Be My Eyes don’t have any control over. I do know however that their legal expert is meeting with Open AI and having ongoing talks with them to try and find a resolution that allows us access to face information. Honestly BME are really pushing hard as they seem to understand exactly how important face descriptions are. If theres a way, I’m sure they’ll find it. Troy may also be right about the times we’re living in. I have raised the valid opinions about censorship and adult content on the forum so at least BME will be aware of what people think and what they’d like to see in future versions if its possible.

They just need to make it so whatever face we trying to scan and get a describtion, it does not save the data.

I agree with Holger Fiallo on all points. More if people wouldn't be so damn sensative this would be a much different experience. If we could see perfectly fine we would have access to this. This why we all need to submit feedback to Apple. Apple would fight for us.

One thing we need to keep in mind is this is a new technology. It is very fluid at the moment and is constantly changing. I've been beta testing for almost 2 months and have seen many changes in that time. For example, when I first started testing I was able to get the AI to read bank card and credit card info. It will no longer do this. I don't know when or if that functionality will ever return, but I bring it up just to illustrate how things are constantly changing. We need to provide as much feedback as we can and let them know what we want this technology to do for us. I know that BeMyEyes is doing everything in their power to make this the best it can possibly be.

I get the frustration but we are using technology to “see” the world and laws are laws, people have rights and there is no way to sugarcoat the facts. It’s not fair but it it is what it is. I’m not going to waste my energy on trying to make that happen when the reality slapped me and said get over it & move on to more things that I care about. That battle was already fought and we lost it and complaining isn’t going to win it. The company who processes the images we send has a reasonable decision to keep them to make AI smarter and better for us for accuracy so that’s a good thing for us. I’m not comfortable with them storing my face in their database and you shouldn’t be either.
Sighted people do not belong in a discussion about tech apps that blind/V I people use to process images… that’s comparing oranges to apples.

I'm so glad you don't hold any decision-making positions! With this attitude, nothing will change for the better. And, unlike you, I don't think that the company which processes the images we send has a good rationale for not interpreting faces or refusing to read written material this way.

I've also seen this a good number of times and wonder how it makes such discretionary decisions when, in fact, nothing is wrong with the text on the banner or poster. Honestly - and as things stand right now, I don't trust this technology much, and it is my last option when it comes to understanding my surroundings with facial features removed and such a high chance of hallucination and misinterpretation. And, yes, I still love and admire their provision of access to trusty human volunteers. From the usability/practicality standpoint, I don't care which laws or regulations limit its functionality. What I want is better OCR capabilities, no face-related removals, and less hallucination.

This link points to a picture which contains offensive language.

When I ask Be My AI to describe it, it initially omits the most offensive message, only mentioning that there is an offensive message, but it is happy to transcribe the content of the message when I ask what it says.

Because of the politically charged nature of the image, I would like to say as a disclaimer that I am only linking to it for illustrative purposes. Please do not take my linking to the image as an endorsement of the offensive slogans contained therein.

I agree that censorship should be up to the individual and I’ve raised it with BME. As I said earlier, OCR is most definitely on the way which should help significantly with BMAI making things up which will be a huge improvement. Faces are sadly just one of those things we’re going to have to cross our fingers and wait and see. This is such a new technology and it allows us to do and know things that we’ve never had access to without sighted assistance before. Because it’s such a new technology and the entire world is interested in it, it’s going to take some time for everything to get worked out. I always remember, we’re at the razors edge of a technology thats almost certainly going to change the course of human history. Thats a pretty big statement but it’s true. Having a level of inteligence and understanding not inside a human brain is going to take some working out. Personally I don’t think faces are a privacy issue because all of our faces are in thousands of records all over the world. That ship has already sailed but others will have different opinions and they’ll use the law to try to make the world how they want it to be. The unfortunate side effect is that we can’t find out whats on peoples faces for now. It’ sucks but this issue is so much bigger than us wanting to and having justification in understanding faces and what they look like. The whole world is court up in this in various ways and a lot of them have lawyers. I personally think this stuff will be sorted out in time, the promise of this technology is too great and theres too much money to be made but time will tell. For now I just remember I’ve got technology and ability I have never had before and its free. Those things are worth remembering IMHO.

Are we children that we need to be protective by those who make apps? I am an adult and if I want data I do not care for anyone to tell me is offensive. It will be my decisionto know. to make

I agree with you, I should be able to decide whats okay for me but I think they are seeing it from a slightly different perspective which is that they are a company with investors doing something thats never been possible before. To try a reducto ad obserbum (forgive spelling) The picture you’ve sent is of a scene thats likely to be pornography. There is a female looking a bit withdrawn and sad while trying to put on a brave face for the camera. There are 8 men around her in various poses. 2 of the men appear to have their.. I’ll let you work out the rest of that comical description but it kind of makes the point doesn’t it? Where is the line of whats ok to describe. For myself, I’d like everything thats visible to be describable but thats just not realistic in a commercial product. I just appreciate the major step forward I just got.

In my opinion, the decision whether or not to describe potentially objectionable content should be made by the user via setting in the app.

I would hope that whether a user is deemed the right of reading whatever material he/she wants to should be a setting similar to what Apple has in the phone. Sensitive material is turned off on my phone and I left it that way. However, maybe it’s one of those age-related settings.

If you want to read the banner someone mentioned earlier, let Seeing AI or the camera app do its thing. Trust me, there is more than one way to skin a cat!

Hello.
I finally had access this morning, and here are some interesting points about what I think so far.
1.
AI seems to work much better with photos from outside the app because, after testing taking a photo directly within the app, it took an infinite amount of time to process and did not give me any results.
2.
It takes a while to process the images, but in return, the response time to our questions is surprisingly fast, which surprised me when I tried it. Speaking of which, it is much more detailed than Bing's responses.
3.
For non-native English speakers like me, it's worth noting that the AI only writes in English, even if you write the message in another language, it responds in English.
By the way, can you help me? I created an activity for the Be My Eyes app to be in English with the voice of Alex, and I set the app language to English, but the voice doesn't change when I open the app. When I share images for the "describe with Be My Eyes" option, it has the text in Portuguese, and supposedly the app is in English. What should I do to resolve this?
Best regards.

I tried taking pictures of a juce carten and other things but it doesn't read them as well as seeing AI. I could have just had bad lighting but it's a bit sad when this new tech doesn't perform as well as the old.

I honestly can't really see a reason for me to use this app, the novel of "looking" around is interesting but that's about it for now.

Maybe i'll come across a reason to use it in the future, who knows.

Now if they came out with an AI that worked for the blind with maps, describing roots in great details and stuff like that; I'd be very interested. I think I'm looking for a live AI feed like thing, I think I'll come across it one day.

First of all I have found that the AI can understand and can write in my native language. I just told it to speak in Greek and it did.
I have also found that the description it gives to pictures is very very detailed. In fact it describe dust and bubbles on the surface of a pool for me.
It also gave me directions for how to reach a glass of water on a table for example, or for how to navigate from the point the picture was taken to another point in the room. I don’t recommend this Way of navigation because it is quite risky especially outdoors, but if push comes to shove it can be done.
I also had the AI read me all my old PS2 titles from back in the day where I could stil play them, which were arranged on the shelf and it did. I jumped into a pool of nostalgia with this one actually, it was quite moving.
There are also many many other ways in which you can put the app at use, with the imagination of the user being the only limit. For example I had it tell me where are the light switches in the room, where is the door and the door handle, how many people were in a restaurant and if there were any empty tables, what is in my fridge and what can I cook with the stuff, where is my air condition controller which I couldn’t find and I was freezing and many many other things.
To tell the truth I have never been more impressed by a piece of technology. It is superb and the use case scenarios are virtually unlimited. you just need to ask the right questions in the right way.
However, having said that, this is not an application to read large bodies of text. It won’t do it. It will tell you a summary of the text and that’s all. There are other applications for that use. On the other hand it will read you a restaurant menu and it will even categorize what it’s looking at. For example main courses, desserts etc. You can even tell it to tell you if there are any dishes with this or that. For example I fed it a picture of a menu and I asked it to tell me all dishes with eggs and it did.
That’s my experience so far guys. As I said, I am extremely impressed with this application and I have found it to be great help in my daily routine. I can’t wait to go to the supermarket with my girlfriend so I can take pictures of the shelves and, at long last, no what’s for sale, without having to burden someone else to give me the information.

As above, OCR is something Be My Eyes are actively working on. My guess is it’s not far away. As for usefulness, Panais got it exactly right IMHO the more creative you are with your ideas of what to use it for and the questions you ask, the more Be My AI will impress. Remember everyone has been waiting for this for so long and in its current form, it’s imperfect. Faces and OCR are 2 major problems so it’s understandable some people won’t be as impressed as they might have been. It really will give a lot of information though especially if you get creative with how you use it. Good Luck.

I received access yesterday. It's a great step forward.

I have a folder with pockets for documents. I opened it and laid it flat on the floor. The app described it correctly, telling me the heading for the documents on the left and on the right. I told it read me the document on the left, and it read it to me. This is exactly the kind of functionality I want with menus in a restaurant - give me the summary by reading the headings of the sections, and only give me more information if I ask for it. Like a human assistant would do.

I gave it a photo of my washing machine and asked it what I would need to do to wash a large load on cold. It told me which knobs to turn and which way to turn them.

Yes, it has problems with some text. In a photo of my back patio, it identified an analog dial thermometer. I asked it what the temperature was. It announced 62 Fahrenheit and my guess is that the correct answer should have been 82. But even Seeing AI gets this stuff incorrect sometimes.

This is amazing technology. I can't wait to see where it's at in ten or twenty years.

Apple, please replace SIRI with a smart assistant based on current AI. Give me the ability to tell it what I want done on my iPhone or computer. Let me tell it to fill out a webform, delete files, interface with apps I already have on my phone to read me information about the thermostat setting or when the next bus will arrive. Quit playing catch-up and make the future happen.

OK, these use cases are amazing!

I'm getting a new hoover/mop soon so I'll try it out on that.

As of the latest update this morning, the app no longer accepts photos shared with it via the share sheet on facebook. I was going through my friends and family's photos now that I can accessibly know what they look like and when I tried to this morning, I got the message that facebook access with be my eyes is no longer permitted. the app only works with photos taken from within the app itself or the shared via photos.
While I can still just off load a photo to my photos folder on my iPhone, the process to get a photo discription has now become significantly more inconvenient. Be my eyes, do you have an explanation as to why facebook access has been completely revoked?