Update 19 May, 2016: Facebook has today announced that automatic alt text is now available globally in English.
*** Facebook has today officially launched it’s much anticipated tool for automatically generating descriptions of photos posted to its site.
Called Automatic Alternative Text (AAT), the tool has been developed by Facebook’s Accessibility Team and uses object recognition technology to identify the contents of a photo. With photos such a core part of Facebook, this could transform the user experience for blind and low vision users, giving them the opportunity to know what others are liking and sharing.
According to a report by TechCrunch:
The object recognition powering Facebook’s AAT is based on a neural network with billions of parameters, and one that is trained with millions of examples. Neural networks are one type of model for machine learning. When it comes to images, you can think of a neural network as a pattern recognition system. In Facebook’s technology for AAT, it recognizes images and words in transportation (“car,” “boat,” “motorcycle,” etc.), nature (“outdoor,” “mountain,” “wave,” “sun,” “grass,” etc.), sports (‘tennis,” “swimming,” ‘stadium,” etc.) food (“ice cream,” “sushi,” “dessert,” etc.) and descriptive words for appearance (“baby,” “eyeglasses,” “smiling,” “jewelry,” “selfie,” etc.).
In a reply posted below, Matt King from Facebook’s Accessibility Team says that they are “… currently limiting the number of concepts to a small set, fewer than 100, where the system has been very well trained. Naturally, this number will grow over time. And, the accessibility team can provide input to the AI team regarding which concepts we think should have a special training focus.”
In a second reply, Matt states that recognition of text in images is also in Facebook’s plans for this tool:
The ability to recognize text is part of the back-end system, but we do not believe the results are yet good enough to turn it on for all images. We are surfacing the fact that the image may include text. And, when auto alt text is available in desktop and laptop browsers, people with the ability to perform OCR on those images could choose to do so. You should know, however, that we want to turn on Facebook OCR for all images that contain text as soon as we have enough confidence in the feature. At this time, I can't venture a guess as to when that will happen.
The BBC reports that “… Facebook would like the system to go one step further and use face recognition to identify people in a picture by name with help from their database of users, but others are resisting the idea on privacy grounds.”
AAT will initially only be available to users in the United States, Canada, the United Kingdom, Australia and New Zealand; and who are using English as their language. The tool should become available globally in a few weeks to anybody else using English language. Work on supporting other languages has already begun, with Facebook expecting to be able to announce more information soon.
For now, AAT is only available on the iOS app; with support for Android and Facebook’s website expected to be rolled out in the next few months.
If you are using English in one of the currently supported countries but not yet seeing descriptions being included with photos, it’s been reported that simply closing the Facebook app via the App Switcher and then re-opening it may get things working properly for you. However, Matt King has said below in the comments that this should not be necessary.
For more information about Facebook's Automatic Alt Text feature, check out this AppleVis Extra Interview with Matt King, Accessibility Specialist in UI Engineering at Facebook.
We would love to hear your personal experiences and views of this new tool, so please post a comment below.