Facebook Now Providing Automatically Generated Photo Descriptions to Blind and Low Vision Users on iOS

Member of the AppleVis Editorial Team

Update 19 May, 2016: Facebook has today announced that automatic alt text is now available globally in English.

*** Facebook has today officially launched it’s much anticipated tool for automatically generating descriptions of photos posted to its site.

Called Automatic Alternative Text (AAT), the tool has been developed by Facebook’s Accessibility Team and uses object recognition technology to identify the contents of a photo. With photos such a core part of Facebook, this could transform the user experience for blind and low vision users, giving them the opportunity to know what others are liking and sharing.

According to a report by TechCrunch:

The object recognition powering Facebook’s AAT is based on a neural network with billions of parameters, and one that is trained with millions of examples. Neural networks are one type of model for machine learning. When it comes to images, you can think of a neural network as a pattern recognition system. In Facebook’s technology for AAT, it recognizes images and words in transportation (“car,” “boat,” “motorcycle,” etc.), nature (“outdoor,” “mountain,” “wave,” “sun,” “grass,” etc.), sports (‘tennis,” “swimming,” ‘stadium,” etc.) food (“ice cream,” “sushi,” “dessert,” etc.) and descriptive words for appearance (“baby,” “eyeglasses,” “smiling,” “jewelry,” “selfie,” etc.).

In a reply posted below, Matt King from Facebook’s Accessibility Team says that they are “… currently limiting the number of concepts to a small set, fewer than 100, where the system has been very well trained. Naturally, this number will grow over time. And, the accessibility team can provide input to the AI team regarding which concepts we think should have a special training focus.”

In a second reply, Matt states that recognition of text in images is also in Facebook’s plans for this tool:

The ability to recognize text is part of the back-end system, but we do not believe the results are yet good enough to turn it on for all images. We are surfacing the fact that the image may include text. And, when auto alt text is available in desktop and laptop browsers, people with the ability to perform OCR on those images could choose to do so. You should know, however, that we want to turn on Facebook OCR for all images that contain text as soon as we have enough confidence in the feature. At this time, I can't venture a guess as to when that will happen.

The BBC reports that “… Facebook would like the system to go one step further and use face recognition to identify people in a picture by name with help from their database of users, but others are resisting the idea on privacy grounds.”

AAT will initially only be available to users in the United States, Canada, the United Kingdom, Australia and New Zealand; and who are using English as their language. The tool should become available globally in a few weeks to anybody else using English language. Work on supporting other languages has already begun, with Facebook expecting to be able to announce more information soon.

For now, AAT is only available on the iOS app; with support for Android and Facebook’s website expected to be rolled out in the next few months.

If you are using English in one of the currently supported countries but not yet seeing descriptions being included with photos, it’s been reported that simply closing the Facebook app via the App Switcher and then re-opening it may get things working properly for you. However, Matt King has said below in the comments that this should not be necessary.

For more information about Facebook's Automatic Alt Text feature, check out this AppleVis Extra Interview with Matt King, Accessibility Specialist in UI Engineering at Facebook.

We would love to hear your personal experiences and views of this new tool, so please post a comment below.

Blog Tags: 

61 Comments

#1 Excited

This is the sort of thing that makes me feel excited. I deleted my Facebook account because too many photos without sufficient descriptions were appearing in my newsfeed. I may consider getting a new account to try this out. Does this work on the Facebook website?

#2 Not working?

How please, does this work? for me, if someone has put up 4 photos i still just hear photo photo photo etc. what am i doing wrong? even clicking on the photo just saying image? i am set to english using the facebook app in ios 9,3,1 thanks to all, William.

#3 I can't see how to do this

I don't have any option allowing me to do this. I'm running the latest versions of both IOS and Facebook as far as I can tell. Only thing I'm thinking is that I am set to English UK. Do they in fact mean this will only work with US English?

#4 Not working here either.

Probably a release bug.

#5 set language to united states

It does work, in response to bingos post, set the actual language of your phone to u.s. and region to u.s. worked for me. whilst wont describe every nuance of a photo at least we get an idea of what a photo is. it cool!

#6 AAT

I wonder how many decades it will take until it finally reaches Sweden? Shall we take a guess at around 2040?

#7 Still not working.

I guess one must actually live in the US.

#8 Try Restarting the App

Member of the AppleVis Editorial Team

While I can't say whether this works for other languages besides English and other regions besides the United States, I did have to close the app in App Switcher and re-open it for the Automatic Alt Text to start appearing.

#9 English only

It's clearly stated in the blog post that for now this feature is only available for english language, hense my maybe somewhat sarcastic statement.

#10 Works

Just tried it a few minutes ago, took me a while to find a pot with a picture, everyone is sleeping in my part of the world, but I found it, and while its not the very discriptive things from TapTapSee, its enough for you to know whats in the picture. Close and open the app from the app switcher, or just weigt. I think I heard somewhere that its coming to other english speeking contries. Maybe its a rollout feature.

#11 Facebook Photo Descriptions

I live in the United States. My iPhone and Facebook Settings are set to English. I am running the latest versions of IOS and Facebook. I have powered my phone off and back on. I have closed Facebook and re-opened it. And I did just find a photo a friend posted day before yesterday. VoiceOver says, "Image may contain two people smiling." On the other hand, Seedlings Braille Books for Children just added five new photos to their Facebook page. On four out of five, VoiceOver says, "Image may contain text." On the fifth, VO says "No automatic alt-text available."

#12 Thanks

Thanks for confirming my suspicions. Only available in US English I believe would be more accurate. Goodness knows what Facebook think our official language over here is! Thanks for the suggestion re changing my language and region but I don't want to do that just for the sake of Facebook. I'll wait for it to come to me rather than going to it. Completely sympathise with the comment re Sweden. it seems that some countries are always last and I would be very aggrieved if I were you.

#13 Photo discriptions

Its cool!!!! It describes how many people are in the photo. Even says if its a selfie lol. I'm wondering if it's supposed to read the text though? It says "contains text" but doesn't read it. Also what is alt text? Sometimes it says "no alt text available". Or how do I get it to read the text on the photo?

#14 Not working here

I'm in South Africa, checked for any Facebook updates, closed the Facebook app and reopened it again, but just said image.
Then I changed my region to US and language to English US, closed and reopened the app again but still no luck.
I hope it's a rollout thing.

#15 I must have been a beta tester

I have had this feature for a couple of weeks. Facebook must have opted me in. I also had a survey request and I filled it out. So far, I like it, but, it has its limitations.

In my experience, and noted in my survey response, I would like to have the text in a photo read. If it can tell there is text, then it would be nice to have it OCR'd.

I have not found it to be overly specific. For exampl,e I will hear things like "four people smiling" or, "Food on a plate." I wish it identified the people, but I'm probably good not knowing what the food is...

#16 Hunky-dory in the UK! :)

Hiya, it's working perfectly here in the UK… I have set my language to British English!
Cheers,
Shannon

#17 Reading text

I would like to have text read to. Especially with all these memes people post. But this just started. It will get better and better with time.

#18 still not working for me

I'm still determinedly in the UK and it's still not working for me. When did this Facebook update come out? I last updated Facebook on 2nd April or something like that.

#19 Last night people were

Last night people were posting things from the basketball game facebook told me basketball court national champions banner pretty cool.

#20 A possible workaround !

Try closing the Facebook app and opening it again. It worked for me!
Cheers,
Shannon

#21 The facebook aat works in the

The facebook aat works in the uk also , but does not know what a guide dog is just states this image may contain one person , clouds , sky , nature and i am sure i was smiling , a great first step can only improve

#22 Facebook

Hi.

This is something great.

I've used it several times and I now feel like I know what people are liking etc.

The feature is in its very early stages so it can only get better.

I read on the BBC that facebook would like to try and match photos from people who use facebook and try and tell us who is in the photo. how cool is that?

Bryan

#23 Not working

Not working here either and I'm in the uS. will try to close and reopen the app though.

#24 works, kind of

Well, to me, image may contain text, or image may comtain a plate of food isn't really a description at all, it's more of a guess. But, as everyone else is saying, give it time. This isn't ever going to be perfect, I think some of y'all are expecting too much too soon.

#25 Not working here

My iPod set to English, I also try to restarted the Facebook app but not working. Do I need to change my region also?

#26 Photo descriptions

For those that say it doesn’t work, simply closing and opening Facebook may not work. They have said that the feature will be rolled out over the next two months and so all might not get it right away. Don’t worry, it will happen. I had it from the first day but my fiancé got it several weeks later.

#27 Thank you

So I'll have to wait

#28 think of it this way

For you guys who have to wait for this functionality, by the time you get it it might be even better than it is now. Like I say it's more info than we've had before and it's only getting started so hang tight. It'll work for everyone eventually.

#29 It works

Closing and reopening the app didn't do it. I had to empty the app switcher and then reboot the iPhone. Now I have it, I'd say it's a start. The descriptions are not very informative really, but they are better than nothing. As a beginning it's not bad.

#30 I'm Excited

I'm excited to try out this feature when it comes to the Mac. I can't try it out at this time, as I don't own an iOS device. But this is so cool and those of you who haven't listened to the podcast accompanying this blog post should do so.

#31 I've had this feature for a while now

Hi,
I don't know why but I have gotten this feature from a previous FB update last month. I was not beta testing FB so I don't know what happened there.
It's an ok feature. Let's give it time to see if it improves.

#32 Another interview with Matt King

You can hear another interview with Matt King from the Facebook Accessibility team about this new feature as well as some demos by listening to this week's episode of Eyes On Success (to be posted 8:00 a.m. eastern time on Wednesday, April 6):

1615 Accessibility of Facebook Products (Apr. 6, 2016)
Do you struggle using Facebook with a screen reader? Matt King, a member of the Accessibility Engineering Team at Facebook, describes many tips and hotkeys for using Facebook more efficiently. Hosts Nancy and Peter Torpey also talk with him about an exciting new feature launching this week to help make photos more accessible to people with vision loss.

The show can be heard or downloaded from:
www.EyesOnSuccess.net

and is also available on iTunes.

#33 Cool!

It's working for me wish it was more descriptive. Neat though!

#34 Photo may contain food

Great what kind of food. lol

#35 Not working

Looks like I'm one of the unlucky ones who has to wait for it to be rolled out. I'm in the US, and have restarted the app, my iPhone, and uninstalled and reinstalled the app. Nothing yet.

#36 I experienced it last night.

I experienced it last night. It was saying 'image looks like, cat".
And then it was trying to differentiate between in and outdoors, I'm guessing as in if the photo was taken in or outdoors.
It does need to improve but I was surprised to all of a sudden get the extra information since I didn't know it had been released yesterday.

#37 Automatic alt text on Facebook web sites

App Developer

TJT, the current launch of automatic alt text is for the IOS app. The Android app will be next followed by the Facebook web sites. I don't have set dates for each yet, but if all goes well, they will all be rolled out in the next month or two.

#38 Automatic alt text in more languages and countries

App Developer

Krister, The English version will be available globally in a few weeks or less. Today it is in United States, Canada, United Kingdom, Australia, and New Zealand. Work for more languages has already begun and we should know dates availability soon.

You don't have to update your app to make it work. When it is available in your location it will just start appearing in your feed.

#39 When it says "No automatic alt text available"

App Developer

If you hear "No automatic alt text available", that means that the Facebook object recognition engine attempted to analyze the image but was not confident enough about what it found to provide any alt text.

All the words you hear after the phrase "Image may contain" are what we refer to as concepts. We assign a minimum acceptable confidence level, AKA precision, to each concept. There are some concepts where we require a very high level of precision, say above0.95 on a scale of 0 to 1, and other concepts where we allow lower precision, e.g., 0.80.

If none of the concepts recognized by the system meet their minimum threshhold, we will generate the phrase "No automatic alt text available."

Further, we are currently limiting the number of concepts to a small set, fewer than 100, where the system has been very well trained. Naturally, this number will grow over time. And, the accessibility team can provide input to the AI team regarding which concepts we think should have a special training focus.

#40 Recognizing text in images

App Developer

One of the most critical decisions on this project was to decide when to go live -- when was it good enough?

At Facebook, we lean toward some value is better than no value and sooner is better than later. That said, going live too soon can be disasterous.

We prefer data-driven decisions so we conducted some user research and studies. The data seem to indicate that we could be providing value even with a relatively small number of concepts included in the automatic alt text, and even without text recognition, even though text in images is very common.

The ability to recognize text is part of the back-end system, but we do not believe the results are yet good enough to turn it on for all images. We are surfacing the fact that the image may include text. And, when auto alt text is available in desktop and laptop browsers, people with the ability to perform OCR on those images could choose to do so. You should know, however, that we want to turn on Facebook OCR for all images that contain text as soon as we have enough confidence in the feature. At this time, I can't venture a guess as to when that will happen.

Yesterday's launch was only the first of many steps down a long, new path. It is kind of like 30 years ago when OCR required a perfectly clean, flat page with crystal-clear block fonts. In a few years, or sooner, what we have today will seem like child's play. For now, we would like to step, learn, iterate, step, learn, iterate, etc. so we can make the future happen as fast as possible.

#41 Got it now!!

Woohoo, I've got the feature now. I noticed it this morning!

#42 Working For Me Now in the UK

For people it's not working for, you might try going to the updates tab of the app store and making sure the device has downloaded the latest version of the Facebook app. It's working for me now in the UK with UK region settings. Most of the photos just have "Image may contain x number of people" but I guess that's what mostly people take pictures of. I have had some photos with descriptions of water and sky and food and drink and articles of clothing. Definitley a positive step in the Facebook experience for blind users.

#43 Coole

Glad to hear there will be further updates. Perhaps to get around the privacy issue of saying people's names in posts to tell you who the person is there could be a prompt that people could enable or disable it? I say this because it would be neat to hear doing such and such. Or perhaps it will only say the name if your friends with the person just some thoughts.

#44 Portugal and brazil.

Hi Matt.
congratulations for this great work.
Matt, please, don't forget Portugal, and brazil.

make this feature available to Portugal and brazil.

thanks Matt. cheers.

#45 I CAN'T WAIT FOR IT TO BE ROLLED OUT IN SOUTH AFRICA

I hope it gets rolled out in South Africa too. Can't wait.

#46 Automated descriptions and human descriptions working together?

Hi, On the off-chance that a developer will see this: I sent a long email to Facebook accessibility about the possibility of the automated image descriptions being paired with a mechanism for human-generated image descriptions. It would almost be like Be my eyes, except the user could just click on a button next to the image and request more information. Also, unlike Be My Eyes, I think that blind people should form relationships with specific sighted describers, or at least have the image described by someone in their demographic, so that there's a better chance of salient details being captured. If anyone would be interested in developing something like this, I'd love to help as the idea girl (I can't code, unfortunately). Tasha dot chemel at gmail.

#47 About human descriptions...

Hello all,
Reading the comment about human and automated descriptions together made me think of this: Why can't I add captions on my phone when posting a photo?
Whenever I took a picture and wanted to add it to my Facebook status using the Facebook app, there was a text field to add a caption to the photo so that VO wouldn't just say, "Photo." This was different from the text field labeled "Say something about this photo...".
Captions can still be added on the Facebook website, but I can no longer add captions using my phone, which is where I do 99% of my Facebook activity.
Any help with this would be greatly appreciated. :)
Kira

#48 Human descriptions

Hi everyone, first of all, let me say that I've got the feature working, and I like it so far as a first step. I think we all very much appreciate the efforts in this area! I agree with the comments above, in that it would be nice to have our friends add captions, which become alt text. I use Zoom along with VoiceOver since I have some residual vision, and I have to say that often what I see is a lot more than what is automatically described. I think it would be a great step forward if humans could add descriptions to pictures, and have these descriptions help “train” the computer algorithm. For instance, one of my friends posted a stunning shot of a dolphin in mid-jump. The description said, “Image may contain ocean”, but nothing about a dolphin. If I could have added that, and if enough people added that on other pictures of dolphins, then maybe the neural network could learn what a dolphin looks like. Just a few thoughts. And again, this is only a suggestion for how we can add details in the future. This is all very exciting!

#49 One thing I'd like to see the

One thing I'd like to see the accessibility team at fb work on as well is when someone sends you a sticker right now vo will just say sticker, it would be nice to know what that sticker is.

#50 Why not just reactivate

Why not just reactivate your old facebook account instead of having to create an new one? That would be much easier instead of going through so much information. To make your new account you would have to add your birthday, email address, new password and so on. Just reactivate your old account again. You do that by loggin in with your email address and password. Or did you delete your facebook account permenantly?

Pages