In this episode of our podcast, Thomas Domville demonstrates the power of the Virtual Volunteer feature, set to come to the Be My Eyes app in late Q3 2023. Currently in beta testing, this feature, powered by OpenAI's GPT-4 model, has the potential to be a game changer for people with visual impairments. It offers a virtual sighted assistant that can generate context and understanding for images, allowing for a greater degree of independence in everyday tasks.
During the episode, Thomas showcases a variety of real-world use cases for the Virtual Volunteer, including identifying clothing; getting information from food packaging; describing greeting cards, photos from your photo library or places such as Facebook, and weather maps; reading restaurant menus, and more.
We thank the Be My Eyes team for allowing us to record and share this demonstration of the Virtual Volunteer.
Comments
Yeah, definitely something has changed again
Now, when I take a picture that has people on there, be my AI will say something like this:
I apologize, but it seems that the image you've uploaded was blocked because it may contain faces or people. For privacy and security reasons, I'm unable to view or describe images that contain faces or people. If you need assistance with something else, feel free to let me know or upload a different image.
I hope this will be fixed as soon as next week
a question
Like the rest of you, I'm definitely looking forward to having this fixed soon. Also, I think I already know the answer to this, but is there any way to get email updates whenever this thread updates? I know that's a thing with forum posts, but from what I understand is not possible with podcasts.
Screenshot of dating profile worked, but
It’s disturbing to me how I took a screenshot of a conversation I was having on Facebook dating, some of this person‘s photos are in that conversation so I took a screenshot, and it described the face on be my AI but when I went to my photo album and tried to do that from the share sheet with a normal photo it won’t describe it and it came up with that message like the person mentioned above so then I tried to take a screenshot of this same photo and do it that way, but it would not describe it so it’s definitely buggy!
I’m frustrated about this, I sent them an email last night explaining the issues with this. They need to work on having be my AI Like it was in this podcast a few months ago because it’s not fair to us that we had to hear this wonderful demonstration, and now we have to go through this…
This is what I mean…
The picture you've shared is a screenshot of a dating profile editing page on a mobile phone. At the top, there are options to "Edit Profile" and "Preview Profile". Below that, there is a section titled "Photos and prompts" with text that says "Drag and drop photos and prompts in the order you'd like them to appear."
There are several tiles with photos and prompts. The first row has three tiles. The first tile is a photo of a man taking a mirror selfie showing his upper body. The second tile is a photo of the same man in a gym. The third tile has a prompt that says "I'm looking for someone...".
The second row has three tiles as well. The first tile has a prompt that says "One movie I can watch over and o...". The second tile has a prompt that says "The 3 words that best describe...". The third tile has a prompt that says "My idea of a perfect day is...".
The third row has two tiles. The first tile is a photo of the man holding up two thumbs up. The second tile is a photo of the man flexing his muscles in the gym.
The fourth row has one tile which is a photo of the man shirtless, wearing sweatpants
Well, I would say maybe shirtless would be considered adult material but somehow it’s able to describe that so the inconsistency of this is really what bothers me and I’m sure it does many others.
I appreciate those who are in the works of making this the way it was and hopefully soon. I’m trying to be married by next year and I would like to know what type I’m marrying. Ha!
Seriously, it’s amazing how we can have a tool like this to use independently so we can match ourselves with someone we’re compatible with and know what they look like without having to ask another sighted person. This is a very personal thing that we should be able to do on our own.
You’re gonna need a lawyer I’m afraid
If in one year you’re going to be married to a woman you don’t even know yet. Lol
That’s interesting …
I would like to keep this about be my AI but Iwanted to share that part of my life so I can help make some movement with whoever is looking out on this topic who’s making those moves but I do appreciate your hilarious input. Thanks!
change in messaging
While it still doesn't work, the message that gets displayed when you have a photo of a person has changed. Now, it says that this photo has been blocked because a face or a person was detected in the photo. I suppose this is much more accurate than the previous message.
I'm confused
So, one of my friends just learned how to share photos through the sharesheet and has access to all facial information and nothing is getting blocked. She lives in Canada, but I know the app does not have location access. Is it possible that it's able to do it through wifi? We've tried it with multiple photos and it seems to work just fine for her.
Edit, some photos have gotten blocked, but others not.
Here is one which was described a few minutes ago.
The picture shows three people taking a selfie at night. On the left, there is a woman with glasses and red hair, smiling. In the middle, there is a man with short dark hair, wearing a black and grey checkered shirt, making a funny face. On the right, there is a woman with long dark hair, wearing a leopard print top, looking at the man with a playful expression. They seem to be having a good time. The background is dark with some trees and a wall, indicating they might be outdoors.
This is interesting
I hope this means that there will be fixes soon for this problem. I have tried to describe some photos just now but still the picture is being blocked.
doesn't seem consistent
As I said, it doesn't seem to be consistent but hopefully this is a good sign for us.
That’s because it isn’t consistent
Yeah, I already mentioned that the other day, I just turned off my Wi-Fi, using my cellular data, and then I went to the Be My Eyes settings and turned off location detection or whatever that first setting was about the location and I got 2 images describe with people in it, but the other ones were blocked. It’s not fair to us to have to deal with that privacy issue because we’re blind and we just wanna have access to people’s facial details like everyone else who can see and this app would provide that in amazing detail but that privacy issue is not our fault and being blind is not our fault, but hopefully they make this accommodating to us so we can have a reasonable way to connect with people and have that info. It makes for a good conversation when we have that info like this…
The picture you've shared is of a young boy, possibly in a school photo. He has a warm smile and is wearing a dark blue sweater over a white collared shirt. The background is a simple gradient, mostly bluish-grey. The boy has short hair and a cheerful expression on his face. There is also a small object in the bottom left corner that looks like a clip or pin, but it's not very clear.
location services seemingly irrelevant
I tried disabling location services, switching off wifi, and even changing my location using a vpn but I'm getting the same blocked images situation.
Worth the try I guess
I’m not surprised it didn’t work by doing those since I mentioned how it’s not consistent on which photos with people in it wants to describe. It’s discrimination.
the algorithm is consistent
you guys are hilarious. you think the face policy is inconsistent, when in fact faces have never fully described or accessible. on a good day, it may describe something about the face, but it is never complete. next time when the ai is in a good mood and willing to describe the face, ask the following questions if they are not already answered:
o describe the eyebrows, are they trimmed and well proportioned?
o does she have double eyelid?
o describe her nose.
o describe her ears, are they symmetric?
o does she have full lips, is she showing her teeth?
o how would you describe her chin, is it pointed?
o complexion of her skin?
even with a perfect picture, the ai will not able to answer all of the above questions because parts of the face is always obscured.
i am agnostic about what the ai should and should not do, this has nothing to do with your rights, and don't conflate this sith democracy, freedom and all the bs. as far as i'm concerned, democracy simply means campaigning and voting; right to due process, equality, freedom and whatever you think is great be be found in any types of political system. simply because one can vote doesn't mean he is in charge, nor does it guarantee your "universal rights given by god". alibaba's ai specifically mentions "helpful for the visually impaired" and "describing face", so this is purely technology and people behind it, very little to do with those abstract ideology.
what i really want is i don't want to see people get super excited then super frustrated about something because they perceive something has changed. at least for the facial algorithm, it is very consistent since it opened to the general users.
The Future is Nigh!
All hail our AI Overlords. 🤖🧎🏻♂️🧎🏻♀️
There is a new updates
Be My Eyes, version 5.0.1 has been released on the App Store, but we still cannot describe photos that contain people or faces
LaBoheme
A little over 2 weeks ago, be my AI didn’t allow facial features descriptions but a few days later it did but went back again to different messages about privacy…
Now if you listened to this podcast above, clearly facial features are described by the AI & on YouTube the living blindfully channel did a demo a few months back with the same type of full detailed of facial features so your statement was incorrect on the consistency.
A little over a week ago, I was able to ask the AI certain questions like you had above, and it answered me. 😊
Before the privacy issue…
Here is an example of how it was.
https://youtu.be/U_UDGFUSGwo?si=bBuxcwhsD8n92BE2
Martin, I want to ask you something
What should we do to solve this problem?
Indra, what can be done to fix the problem.
There’s not a huge amount to be done. BME and Open AI are working on a permanent solution which they are hoping to have working before general release. They seem pretty confident this is going to happen. They are both extremely committed to getting facial descriptions working but the law is a difficult thing to work around. They believe they have a solution which is geographically isolate the places where legislation disallows facial descriptions so the rest of the world is able to use them but this isn’t something that can be reliably implemented overnight and it has to be reliable as its compliance with the law and that could lead to legal action or fines. I’m extremely reassured that the best people possible are on this and they will find a solution. I know some of them and their commitment to this issue and the community in general is beyond question.
Thank you for your brief explanation
So now all we can do is hope and pray. Hopefully this program will run well, and when it is released to the public later, we can take advantage of all the features in be my ai better. Thank you andy.
Be My Eyes just published an…
Be My Eyes just published an update on the subject of faces:
https://www.bemyeyes.com/blog/image-to-text-ai-and-blurred-faces-whats-going-on
A legit response from Be My Eyes this morning
Appreciate that notice. Just here to share.
https://www.bemyeyes.com/blog/image-to-text-ai-and-blurred-faces-whats-going-on?fbclid=IwAR0jH7cCOh8g60SRZ35PdSC2h8IvGAThz3dXpcIxAFAVD_QqQs6MjsGx_Vo_aem_AXcUchnYOdkaoQ3YJJqk5S9B-yeN_d7Fvc7om3fMe_7I1GKU9Dg_RLzIoALChD3QEpo
Keep making statements about how inconvenient and unfair that la
The only thing we can do is keep pressing on the issue that those privacy laws are invalid for our personal reasons of having this technology. Clearly, be my eyes is in developing this feature for us, but they know we are frustrated and need the same type of information that everyone else has who can see and has eyesight. They read these comments and they know that we are dealing with this issue personally. The more we make a fuss about it, the more they hear us. We sent them emails, we talk about it on here and they know, they’re working on it. That’s all we can do because laws are complicated.
We have to fight for our rights… to party!
Just Read The Response
Hi again everyone. Subject says it all. I just opened up Be My Eyes and read the blog post from them regarding the facial recognition. Tbh I don't really care about this for now as I'm rather new to the app anyway, but thank you BME team for clarifying things. As has been stated, this is a complicated issue and it will eventually be resolved. It will certainly take time, but rest assured we have not been forgotten. I for one am incredibly grateful for this app. I haven't done a whole lot with it yet, but that will definitely change in the near future as I *hope* to get a phone with better battery life and memory. I've shown Be My Eyes to at least one neighbor who is very impressed. Another neighbor knows about the app but has never seen it.
Be my AI
I love using be my AI. I think it's such a useful feature, and be my eyes have done a great job with this feature. I've been testing the feature for a few weeks, and I can truly say, it has changed my life. Whilst it can't tell you about faces, everything I've used it for, the accuracy of it has been amazing. This feature has enabled me to read the game menu on my WWE game on PS5, witch before using this feature, I'd always have to wait for sighted assistance. With this feature, I'm now able to do it on my own. Great podcast, and thank you for making the podcast
In reply to Indra
I'm experiencing the exact same thing, I take a picture of my face, I've even shared a picture from Prince Harry, because I follow Prince Harry on Facebook. And I've tried getting it to describe the picture, and at first I thought it had something to do with what Prince Harry was sending, I literally thought, make no mistake. I literally thought, Prince Harry was showing his nudes. But then, when other people took pictures of their faces, or I did, that's when it give me that sensitive content thing as well. To me, it feels like somebody just came in my sleep and gouge my eyes out.
When this first came out, I enjoyed taking pictures. My boyfriend and I are both blind, and we were both sending pictures of each other, he would send pictures of his adorable haircut to me, I would send pictures of myself at the beach. It was a great time, like every couple sent selfies of themselves, that's what we were doing, and it described everything accurately. But for this to happen, it feels like our eyes have been pulled out of our socket. That's how I see it. I really really do hope they fix this. I'll tell you something, at first I was supposed to photos and I was supposed to be my LA.
I meant, I was very opposed to this, be my AI. I was opposed, and I disagreed with it entirely, but then, when I signed up, and I started taking pictures, it was a game changer, and I all of a sudden apologized for my horrible opinions against it. It really not only, changed my life, it enhanced our relationship. Only for this to happen, it's a disappointment.
In reply to Martin
The similar thing happened to me. Yesterday I try to send a picture of a funnel cake come up to my boyfriend to show him what it looks like, and I could finally say what it looks like, as a blind person. But I try to show him, because, they don't have funnel cake in the UK, so I tried, showing him what it looks like, and, when I took a picture of it, it gave me that sensitive content message, however, when I open the picture on WhatsApp that I sent to my boyfriend, it described it very well. And he even got a description of it very well, so to my relief, he was able to get an idea of the funnel cake. But it's just so weird, how in one app, it'll tell you one thing, but then it tells you that it doesn't work, or this whole sensitive content image blocking placeholder thing.
I really hope they fix this, because this is seriously annoying
in reply to Michael
I think, as of yesterday to me, the message changed as well, saying that they couldn't tell me because of faces, but the image was blocked, because I detected a face or something like that. I still don't see nothing wrong with faces at all, yeah, the message must be better than the previous one, and it's understandable, but I really feel the persons face shouldn't have to be viewed as sensitive content, if it was something else, and that's understandable. But it said that similar thing to me yesterday, when I was trying to send a picture of a funnel cake, or a person holding the funnel cake. Which was me.
holding out hope
Tre,
You put it very well. It's like we were granted our abilities to see then abruptly, it was taken away.
It is very jarring and I also derived tremendous pleasure from being able to appreciate photos. My wife and I were having so much fun going through our old photos of our marriage, honeymoon and vacations together.
descriptions vs identification
I think this blog post raises a good point. There is a difference between descriptions and identification. at the moment it seems like those lines are a bit blurred (pun absolutely intended). This is what gives me hope that it will eventually return as it was before.
From my understanding it's a…
From my understanding it's a battle between be my eyes and gpt chat4, gpt doesn't want it while be my eyes does.
Guys, please help me to confirm this
Just now I try to recognise a Photos that has people on it using Be My Eyes. I used two different photo and the result is good. There is no such thing like placeholder or something. Maybe you can do the same thing like me and post the update and result here. Thank you
And this is my example
The picture shows a group of people standing in front of a restaurant at night. There are five people in the group; two adults and three children. The adult on the left is wearing a white t-shirt and the one on the right is wearing a teal headscarf and a dark-colored shirt. The children are in the middle; one is wearing a maroon shirt, another is wearing a light blue dress, and the third is wearing a red shirt with a grey jacket. Behind them, there is a sign that reads "Gacoan Mie Pedas No. 1" in bright neon lights. There are also cars parked in the background and a "Buka 24 Jam" sign which means "Open 24 hours". The picture has a caption that says "Late post" and there is a timestamp in the top left corner that reads "17.13".
Face descriptions should be back now.
It looks like Be My Eyes have a bypass with open AI and face descriptions should work. Hopefully they will stay this time.
faces
I can confirm that it appears to be working fine here for the time being. I echo the sentiment of the previous poster. I hope this is the long term solution.
Yes, they fixed it!
Yes, they fixed it!
I’m not getting my hopes up again, but happy for now
If they take this away again, I am not going to even attempt to use the app anymore the back-and-forth with us is a crying shame for the company to do that for us and then take it away again. I’m grateful that it’s back but again, if this happens again, I won’t even attempt to do anything with the app with people in photos because I know it’s going to put an upset with me for the reasons that are already out there!!!
It did a wonderful job at telling me what I needed to know from an image on Facebook dating not long ago and want that to stay just like everyone else and it’s not like we don’t have good reasons for that not to stay so we can get the information we need, have good conversations with people, and enhance the quality of our lives by having that information. Thank you for whoever made that happen, but please do not take that away again. I appreciate this. 🙏
Now, to address the people in the comments, yes, when we have the information about peoples faces, we are able to converse with people about relevant information regarding their looks, their facial expressions, clothing or whatever information that is gathered in that photo that we can talk about freely just like anyone else who can see faces Which makes our lives that much more better. And it’s not just about the faces, it’s about the clothes that the people are wearing and the activities they may be doing in that photo. Seriously though, why would someone want to block that from us? If the shoe were on the other foot and sighted people suddenly became blind, and they wanted to have that information, they would feel the exact same way we feel about this technology. It makes absolutely no sense for anyone to block that from blind people having that information to use in a positive manner. And that brings me back to laws, it makes absolutely no difference when it comes to laws because it’s not like we are using that information to do any harm to people or whatever the case might be.
As a matter of fact…
It warms my spirit to see that people are excited about this feature being added back, and when the person above explained the situation with her boyfriend who is in another country, they sharing their lives back-and-forth with the descriptions from be my AI, and the gentleman above explaining going through photos of his wedding, and having that conversation with his wife about their wedding photos, that just made me so ecstatic for their relationship and the mini conversations they will have With this technology. And it brings me back to the reasons why we should be able to have this feature stay and what it means for the quality of our lives. This is not just about blind people talking about images, but what we can do whenever we are out in public, we need to take photos of places so we can get information about whatever we’re trying to find, and in those cases, people might be in that image, and we won’t be able to have that detailed description because of this blockage of faces that happened and that would definitely make our lives harder for us to do without having this technology and not all the time we want to call up a cited volunteer on Be My Eyes to get that information with the live video… we would independently do it on our own which is much more liberating, even though we are using AI technology. Thanks again.