An Open Letter to Apple Regarding Xcode's Accessibility

By mehgcap, 22 October, 2014

Dear Apple:

For years, I have used your products. I took the path that many people take: an iPod Touch, which slowly drew me into the joy of a simple, powerful, and overall great operating system; to a Mac, which slowly replaced my Windows computer thanks to OS X's lack of crashes and errors, syncing with iOS apps, ease of downloading apps from the App Store, and other features; to the iPhone, because it's a more powerful iPod that can get online anywhere and make phone calls, so why would I not want it. The ease of use, power, usefulness, stability, apps, and seamless syncing have done their work, and I am a (mostly)happy Apple user. Oh, one thing before we go on: I'm blind, so I use Apple's wonderful VoiceOver screen reader on iOS and OS X.

Now, in 2014, I am an experienced user of your products and software, and am (mostly) happy with VoiceOver on the mobile and desktop operating systems you produce. I also have a Bachelor's degree in computer science, plus the time, knowledge, and desire to produce iOS and OS X apps, and I have been a casual Xcode user since version 4.5. To date, however, I have not yet released any of my work. The reasons for this are numerous, but the main one is this: Xcode, which is the only way to produce apps for your products, is only mostly accessible, and only partly efficient, for VoiceOver users.

Before we go on, I want to be sure you know this. I am, as I said, a satisfied customer, and I always recommend Apple products to other people. I admire the accessibility work you have done so far, and the strides you take with each new update to your products. I love that I can enable VoiceOver on anyone's Mac or iOS device and instantly know what I'm doing, and I love the independence that comes from being able to update, re-install, and troubleshoot my own computers without sighted help. I love my Mac, and it's fair to say that my iPhone has profoundly changed my life.

I don't say all that to try to butter you up, or to soften the blow. Rather, I say it so you won't see me as some Windows developer who grudgingly got a Mac, found it to be not what I was used to, and now wants to unload on you for not being Windows. I also say it so you will know that my perspective is an experienced user who would love to be able to bring apps to the App Store that will enhance the lives of others just as Apple's own software and hardware have enhanced my own life. I want to help you help us all, and you can't fix what you don't know is broken. I might sound frustrated at times in this letter, and trust me, I am. However, I realize that these things take time, and that Apple is not an accessibility company, but rather a mainstream company who holds universal access as one of its central beliefs. At the end of the day, you have many priorities across dozens of departments and engineering teams, spread over several product lines and hundreds of millions of users. I'm realistic about all this, but from this end user's perspective, I feel like the below issues are in serious need of being addressed.

Interface Builder

IB is the core of the user interface design. Without it, you are hand-coding your interfaces, which is certainly doable, but is far from desirable, for a few reasons:

  • If you share code, or collaborate with others on a project, most of your fellow coders will be using IB and will not want to wade through dozens of lines of code defining something they lay out in seconds with a mouse. If they want to change something in the UI, they must update your code, instead of relying on the IB mechanisms already in place.
  • If you need to look up how to do something on the internet, you'll almost certainly find examples and tutorials using IB, not walking you through the necessary code. In other words, the documentation for coding apps in Xcode is mostly geared toward the use of IB.
  • Debugging is hard, and removing an entire category of potential errors is always good. By hand-coding the UI, you are opening up one more avenue for mistakes and unforeseen problems to come up and bite you later on.
  • Using IB is faster - as mentioned above, dragging items around and setting constraints visually is fast and easy, coding it all by hand is harder. Doable, of course, but not as efficient.

Now that we've established why the ability to use Interface Builder is important, allow me to remind you how the process works for a sighted user: open your .interface file (XIB or Storyboard), find the UI control you want in the Object Library, and drag it onto the canvas. To connect that control to code, open your view controller in an assistant editor, control-drag from the control on the canvas to the part of your code you want, fill out the information in the window that pops up, and hit enter. Simple, intuitive, and extremely well-documented. From official Apple articles to all the tutorials, blogs, and books on the internet that deal with Xcode and UI design, these instructions remain basically the same.

VoiceOver users have a very different experience, however. It was worse in Xcode 5 and earlier; in Xcode 6, more controls now have labels and hints, and I'm guessing that more work on IB's accessibility will be coming in future updates. As things stand now, though, IB is difficult, sometimes impossible, to use.

Adding Controls with VoiceOver

Now, here's how to add UI controls with VoiceOver: open the UI file, find the UI control you want, copy it with cmd-c, and paste it into the outline table, where it may or may not land where you wanted it, or may do nothing at all. This is still pretty simple, of course, assuming it works in the first place. If it fails, find the control in the Object Library, route the mouse to it, and perform a mouse click. Also easy to do, but the problem is that these methods are not documented on any official Apple sites. Therefore, the only way blind developers will find out about them is through sites like AppleVis.com or Maccessibility.net. Following Apple's own tutorials, which are otherwise great for getting started with Xcode and programming, will leave most visually impaired developers frustrated and confused since the instructions for sighted users and those for Voiceover users are not at all similar.

Connecting Controls to Code

Once the controls are in place, it's time to make them do stuff - the button starts the game, the scroll view fills with data, the other button becomes unavailable, all that. If you're sighted, the steps are simple: open your view controller in the Assistant Editor, then control-drag the UI control onto the view controller. A window will appear where you configure the connection; do so, press enter, and you're done.

VoiceOver users have a very different, and more tedious, process here. They must find the control to be connected in the Outline Table of controls, then stop interacting twice, go to and interact with the Inspector Group, select the Connections Inspector, jump to the bottom of the current item and interact with the scroll area there, then locate the button that has the connection type you want (referencing outlet, click action, getting different values, whatever it is). Once on that button, route the mouse to it, disable cursor tracking, lock the mouse button down, and move back to the Source Code Group, Find and interact with the source code of the view controller, move to where you want the code to be in this file, route the mouse there, and unlock the mouse button. Re-enable cursor tracking, fill out the information in the dialog that should appear, and press enter. In Xcode 6, this process is very unreliable, seeming to work only if you type the code beforehand (which misses the point - Xcode should fill it in automatically). Most times, the mouse pointer will not route correctly, and the connection will fail.

Wow, that's a lot of steps, and they don't even work all the time! As a coder, which would you rather do: a single dragging operation, or follow this convoluted list of instructions which may or may not work? Anyone would rather drag, right? While VoiceOver does include its own set of commands for drag/drop operations, any attempt to follow the instructions you'd find in most Xcode tutorials using said commands will fail. Below are the steps one might take while trying to do just that.

  • Find the canvas holding your UI. Move to the item you want to hook to code, and hit vo-comma to start dragging it. Is it a button, checkbox, or other clickable item? If it is, VoiceOver will inform you that it can't drag items that have a click action, so you can't hook up your control that way. Again, this is how most online tutorials will tell you to proceed.
  • Next, find the item in the table of controls, and try to drag it to your view controller. Drag with vo-comma, drag by routing and locking the mouse; try both on the table row, on the edit field of the control's name, and on the disclosure triangle. Nothing will work.
  • Try the Connections Inspector. Go there, vo-comma on the connection type you want (specifically, the button next to it), and vo-period on the part of the source code you want to connect to. No good, VoiceOver reports that the item is no longer available. In other words, it will fail to drag the connection button, but the failure will not happen when you start dragging as it will in the case of the control on your canvas. Instead, it will happen when you try to drop on your view controller's source code.

How, then, are VoiceOver users supposed to use IB at all, let alone quickly and efficiently? Short answer: we don't, because we can't. As is pointed out in Maccessibility's response to this post, VoiceOver users can employ the Trackpad Commander to quickly locate the items with which they need to interact, saving several keystrokes. Hotspots can also be used to save some time, and using both tricks together is a big help. Still, the instructions VoiceOver users must follow are radically different from what they will encounter online, and (as of Xcode 6.0.1) do not always work.

Random Busy Messages

Xcode is a large piece of software, and it has a lot going on most of the time. You might think, then, that having it beachball every so often is normal, and not worth mentioning. I would too, except that it happens fairly regularly to myself and other VoiceOver users and, from reports I've seen on Xcode user lists, to no one else. (For VoiceOver users, "beachball" is used as a verb to describe what happens visually when an app goes "busy".)

Imagine this: you've just added a couple controls to your canvas, and you click the Attributes Inspector to change a few things. Suddenly, as you put your mouse in the edit field for the control's name… Beachball! You wait patiently, but nothing changes, until you hit command-shift-2 to open the Project Organizer window. You then immediately switch back to your project, and everything is working normally again.

This is a very common occurrence for VoiceOver users. Normally in the Navigator or Inspector, but not always, Xcode will go "busy" and won't respond to anything, unless you switch windows (the Project Organizer is the most reliable one to temporarily fix this).

Other inefficiencies exist within Xcode that are not bugs per se, but stand to drastically improve the experience for VoiceOver users if they were addressed.

First, every code file has what VoiceOver calls an Annotations Ruler beside the text field holding the file's contents. This item isn't a ruler, as far as I can tell, but rather a list of any issues Xcode finds in the associated file's code. Each issue is a button complete with the issue message and line number, and most users would probably assume that you can activate a button to go right to the line in question. However, activating any button in the Ruler does nothing at all. You can't interact with the button text to see exactly how something is spelled, or what the punctuation symbol is that Xcode says is missing, because interacting does nothing at all. You can't even copy the text using vo-shift-c to paste it somewhere else and examine it that way. There is a "jump to line" command that you can use to bring up a dialog where you type in the issue's line number to go to that line. Still, if something is a button, and already has full information about the problem it talks about, one would think that activating that button would do something to help you get to or solve the problem.

Next is the Issue Navigator. Select that in the Navigator Group, find an issue in the table, and go to the source code to fix it. Instead of the click that a sighted user would perform to go to the erroneous code, you have to stop interacting two or three times, vo-right to the Source Code Group, interact, find the group holding the code file that has the error, interact, and vo-right to the file's text field. Only then can you address the problem, and if you forget why you came there… Reverse the process, look back at the issue, and then reverse things again to come back to your code. A keystroke to quickly jump from the table of issues to the code file in question and back would be enormously useful. As previously mentioned, using VoiceOver's Hotspot feature helps here, though I've found it to be somewhat unreliable in the past.

Conclusion

Interface Builder is far and away the biggest hinderance for VoiceOver users trying to use Xcode. Drag and drop, even though VoiceOver fully supports it, fails, and hooking UI controls to code is difficult at best. Worse, the instructions used in Apple's own documentation and tutorials cannot be adapted to VoiceOver users, who must instead manage to find the alternative ways of doing things on their own. This goes against the rest of the Mac, where VoiceOver provides equal access to nearly everything in the operating system with little to no changes to any commands a user might be following.

While there are certainly other areas where Xcode needs some accessibility/efficiency work, IB is the worst offender. Things were terrible in Xcode 3.x, better in Xcode 4, still better in 4.5 and 5.x, and have continued to improve in 6. While there is certainly work to be done, this is a promising trend, and I ask you to please complete the journey to full accessibility. Build on what has already been done: allow dragging of controls right from the canvas using VoiceOver commands; indicate the state of a connection; allow dragging of the connection buttons from the Connections Inspector; identify the items in a .xib or .storyboard canvas so they are not "unknown" and so their help tags properly inform users of their purpose and content; bring the instructions VoiceOver users follow in line with what sighted users do, just as most of OS X already does; stop Xcode from locking up when VoiceOver is running. Allow blind programmers to make more apps to help blind Mac and iOS users. Let blind developers enjoy the app creation process just as much as sighted users do by making Xcode and Swift fun and easy tools for everyone, sighted or blind. Let VoiceOver users follow the same instructions as everyone else, instead of hunting through the internet for obscure lists of convoluted steps, or simply giving up in frustration. The fixes so far seem to indicate that work in this area will continue, and I can only hope I'm right in that assumption. I urge you to keep moving in that direction, and I thank you for the work that has been done so far.

Prospective Developers:

If you are thinking about trying your hand at app development on the Mac, and you read the above letter, you may come away discouraged and feeling like it's not worth the bother. Please remember that there are several blind professionals using Xcode successfully already, so it can be done. What I'm calling for is the conclusion of a years-long path Apple has been moving down, where each Xcode release has been more accessible than the last. I hoped to see Xcode 6 be the culmination of this work, finally bringing complete VoiceOver compatibility to Xcode. That hasn't happened yet, and it really needs to, but that isn't to say you can't use Xcode as it is now. You can, but you need to know the tricks and shortcomings first. The fact that you still need to know any special information just to use Xcode as a VoiceOver user is what my letter is meant to address.

Options

Comments

By Jalys Ortiz on Friday, November 7, 2014 - 21:51

I have to give you ten stars for the way this letter is written; one of the main reasons, though, and perhaps most important, is the fact that you were polite. More often than not, people write to Apple and other companies because they're frustrated or unhappy with something, and it really shows, which I'm sure doesn't give the readers of the letter a good impression. Not only did you demonstrate what you wanted to see improvement in, but you also thanked Apple for their work in accessibility. I'm sure that as of late, Apple probably has gotten lots of people bashing them for the "catastrophe" that was iOS 8.0 from an accessibility perspective, and I'm sure it's welcome to see that there are others who are appreciative of the work they do.
Once again, great job!

By Joseph Westhouse on Friday, November 7, 2014 - 21:51

If Apple got more letters as well-thought-out, detailed, and polite as this, I daresay we might see some very positive results. Well done.

My only question is - do you need to tell Apple how sighted users do something, such as using IB? If I was an Apple tech, I feel like I'd read that and go, "Yeah, I know." But on the flip side, I know you're contrasting it with the hoops VO users have to jump through, so maybe the contrast wouldn't be as strong if you didn't highlight the ease with which sighted users can perform the same tasks. Anyway, just thinking out loud.

By Andy B. on Friday, November 7, 2014 - 21:51

We should sign it and send it.

By Jessica Brown on Friday, November 7, 2014 - 21:51

Hi. Have you sent that yet? If not, when do you plan on sending it? You did a good job.

By mehgcap on Friday, November 7, 2014 - 21:51

Thank you, everyone, for the positive feedback. If you have tried Xcode and run into similar problems, I encourage you to share the text and/or link of this post with Apple's accessibility team. Needless to say, I have already done so.

By Alan on Friday, November 7, 2014 - 21:51

Good job, I absolutely agree, and maybe other people should be added to the petition, could you open one (something like change.org), to allow us to support?

And finally, just my opinion... maybe some technical parts of your letter are not necessary, I mean, all those comparisons between Osx and Windows (I promise, you are the most beautiful girl in the world... etc) and those minitutorials about how to... and how to using voiceover. In my experience, shorter is better, and I'm sure they know exactly how their product works, at least for non voiceover users.

Excuse my english and hope this helps!

By Andy B. on Friday, November 7, 2014 - 21:51

It might be worth his point to use his ability to understand the sighted way of doing things. The more we can hold our own, the better.

By mehgcap on Friday, November 7, 2014 - 21:51

Hello,
I put the sighted instructions in for two reasons. First, it serves as a clear contrast to what VoiceOver users need to do to accomplish the same tasks, both in how long it takes to read the steps and in what they entail. Second, most people on the Accessibility team aren't users of Xcode, so this may be new to them as well. I wanted anyone who reads this to come away with the understanding of how much different and more complex the steps for VoiceOver users are, even if the reader doesn't know anything about Xcode first hand.

By splyt on Friday, November 7, 2014 - 21:51

Hey,

If you feel so, add the below excerpt to your text or change it accordingly:

I know you have enabled several students to have a better life, by using Ipads and Macs without extra cost for the excelent, life changing accessibility on them.
Let me make sure you understand that enhancing XCode related accessibility and usability has the potential to make a huge economic positive impact for all the many many blinds who work with Infoermation technology, as software developpers or as programming teachers or coachers. The demand for IT professionals who are IOS and Mac OS programmers / experts is getting higher and higher due to the IOS usage growing in several countries.
So, we ask you to really consider enhancing accessibility in XCode, if possible releasing the enhancements before the next big XCode update.

Marlon

By Steve Holmes on Friday, November 7, 2014 - 21:51

I think you did an excellent job with your open letter. I like the contrast between what a sighted user does and what we have to go through to do comparable tasks.

One favor I might ask is if you get responses from Apple, could you share them with us, perhaps on this thread? I would be most interested in Apple's reaction to this ple. After all, it was very complete. I don't think I could come up with something this elegant. <smile>

By Alan on Friday, November 7, 2014 - 21:51

And, what about starting something like good old campaign of the month? I think we need a strategy to show that many people agree with this letter...

By splyt on Friday, November 7, 2014 - 21:51

Agreed!

Apple must know how uimportant is having XCode accessible. There are lots of work oportunities, but they won't work if you are twice, three times slower than what a sighted coleag is because the IDE is so dificult to use.
With XCode failing to consistently make connections with VO it's even harder because there is not a consistent way of doing things, neither a hard way nor an easy way, hjust a no way at all.
From this point of view XCode 6 has regretted in terms of
accessibility. I am kit sure folks on Apple are not aware of these dificulties, because fixing them should be easy, its easier than implementing accessibility say on a whole operating system or something like. In fact, probably some of the objects that are not dragable at thhe moment probably need to implement some method of some interface, and building keystrokes to set focus on given parts of a window also is not that hard.
For me, for connections, the ideal would be to build a wizard when right clicking in the wanted element in the outline table. This wizard would present a list of connection bound to it and a new connection with the types of connections available,, let one set up the desired one and put info on the clipboard ... and then a simple command v on the wanted source code would do the trick. This shouldn't be hard to implement, given that all elements to query are available in the window tree of XCode at the wizard runtime context.
Marlon

By Andy B. on Friday, November 7, 2014 - 21:51

Chances that Apple will put extra wizards in XCode to satisfy voiceover users is not very good. The best bet is for Apple to allow voiceover users to do things exactly the same way a sighted user does them. We can speculate on the reason why controls in the library are not dragable, but that is for them to figure out.

By splyt on Friday, November 7, 2014 - 21:51

I do agree.

This is just me thinking outloud. Believe me if VO was scriptable enough I would already have implemented it for long.
However, if things are hard enough for them to implement stuff, making wizards might be easier and faster I would think, so this is an option.

By Victor Tsaran on Friday, November 7, 2014 - 21:51

Hi Alex.
Thank you for putting out Xcode accessibility issues in public for discussion. I am not positive if the open letter is the best way to solve the situation, but it opens a dialog and this is worth something!
As a developer you very well know that if there is no bug report(s), there is a very low possibility that Apple engineers will look at the outlined issues. Did you submit any of these through Radar (bug report site)? If yes, did you make sure to mark the issues as those that fit in the "accessibility" category?

By mehgcap on Friday, November 7, 2014 - 21:51

In short: yes. As soon as Xcode6 came out, I realized the IB problems and filed a bug about them. The IB problems are the most important, most everything else is easily dealt with. Please, anyone else affected, open tickets, email Apple Accessibility, and generally get the word out. The more people reporting these issues, the better. If you don't have a developer account, you can't post to Apple's bug reporting site but you can make comments on Apple's feedback site.

By mehgcap on Friday, November 7, 2014 - 21:51

To anyone following this post, know that Apple Accessibility responded to my letter (which I emailed them). They said they passed it to those in charge of development tools, which isn't their standard response. Plus, a word in the message was misspelled, which I take as a sign that it was from a person and not a response template. I remain optimistic that Xcode will keep getting better, and hopefully soon we'll be developing with the best of the sighted world. :)

By splyt on Friday, November 7, 2014 - 21:51

Megcap,

If they try to talk to you, please redirect them if you can to this post.

Thhere are several small and two or three big enhancements we need and it would be great if they listen to us devs either here or uin the list....
For example, going from the library of components to the view and going the oposit way is something tedious and slow.
If they could make sure we could make a cmd + c on the component in the library and do a cmd v to place it ibn the view, at lease you an go pick a push buton and cmd v four times placing four push btns in the viewm, instead of go forth and back four times to put four different buttons.
See, in an corporative environment one has to be fast, at least as fast as or as closely fast as a sighted person ... and we need to ask them to make things as optimized as they can be.
I am not currently a registered Apple developper, but if you happen to talk directly with an enginier that works on XCode please do let us know and if possible to participate.

Thanks...

By KE7ZUM on Friday, November 7, 2014 - 21:51

wow tha's wonderful. I'm not a developer by any means but I'd love to get my feet wet in it just a bit maybe get swift certified one day.

But keep us all posted on the replies you get from apple.