The State of Screen readers in macOS

This website already has a bug tracker, though I don't know how comprehensive it is. I would hope Apple looks at it, but you never know. Someone claiming to be from Apple came on here a while ago and said employees aren't allowed to look at the website during work hours. If true, this is ridiculous and utterly counterproductive! At best, it's someone trolling, but at worst, it demonstrates the problems with Apple's culture. I'm sure the people working on these features want to do all kinds of wonderful things, but can't and/or unwillingly have their priorities shifted because the corporate overlords have their hands tied.

As for creating a screen reader, my understanding is that Apple doesn't document all the APIs and other things required to create a fully functional one. It should technically be possible on macOS, but not any other Apple platform because they're so tightly controlled.

When was the last time Apple did a real update of VO? Yes, they added nice features and fix bugs when they remember to do so. With windows NVDA and JAWS update their software to address bugs and make it more easy to use. Before you jump on me, do like VO but just asking. Do not use MAC but heard stories about it from here.

I think we need a third-party screen reader. And I would pay for it if it was made available. Voiceover is fine in a lot of ways. And I think an additional option could be beneficial.

VO use to be great! Member snow leopard anyone? It started going down hill when Tim took the job and even shown more bugs in later OS upgrades. There are bugs that have existed for a long time that still, has yet to get fixed. My honest opinion and I've been saying this for a long time, VO needs, a complete redesign and code rewrite! I don't care, apple has all the resources to do just, exactly that. Apple don't fix a lot of these bugs and in some cases, gotten worse. Sure, some have been fixed but VO has more bugs than been fixed Lol. Plus I've heard that VO runs on one core only. I don't know if that's still true but look at the times. It's clearly time for an over hall. However, I wouldn't mind a third party screen reader if it came to be, fact I'd welcome it.

I agree with the last poster. I am proud to be stuck in tutorial hell for the last 10 years so my dev skills....... But I do have experience with lots of different screen readers on different oses and voiceover on mac is flawed by design in many areas now. But until this moment... The best we can do is still to report as much as we can, and also acknowledge that it's very exhausting so nobody should feel guilty of not doing enough or not doing at all...

I'm not familiar of how NVDA actually got started since when I went blind all accessibility technology was already quite advanced. However I did begin work on a screen-reader project in the past, and hopefully attracting other developers to help with it was the reason why I ended up creating an account here after lurking for years. Unfortunately despite all the enthusiasm that my initial demo caused at the time, I did not get that much in terms of contributions, and ultimately ended up giving up on that project because I was spending more time reverse engineering macOS, which is something that I can do just fine but don't consider quality time, than progressing with the project. Fortunately my work on that project actually ended up landing me a job last year so not everything was bad.

There are many problems with the accessibility infrastructure on macOS, some resulting from poor design and some resulting from neglect, but overall the consumer side of the accessibility infrastructure is extremely quirky and full of pitfalls, requiring very high maintenance with lots of testing as from my hands-on experience, there's absolutely no way to approach a large number of cases with a blanket implementation to only have to deal with the exceptions because pretty much everything is an exception. Even things as simple as properly announcing and responding to accessibility requests are not implemented properly in some system services, making it unfeasible to casually talk to any process that have an open connection to the window server without maintaining a blacklist of services that do not behave properly, as the way the accessibility API works makes it impossible to avoid the resulting negative side effects.

Last Saturday, while checking out some gear from the on-line Apple Store, I casually followed a link to a page where people can apply to work at Apple, and for the first time n my life I actually gave it a serious thought, because as a Mac power user who absolutely depends on accessibility to use computers, and as a developer with a lot of easily provable experience with the accessibility infrastructure on macOS among many other fields in software engineering, a perfectly clean criminal and financial record, and a long relationship with Apple that has always been in good standing, I believe that I could actually provide value by helping turn this ship around as an insider. However I am already invested in other options, like creating my own company, and am still working for the people who hired me last year, plus even if somehow I got through Apple's hiring process, that would still require relocating to a different country, and I have absolutely no idea of how much corporate management friction I would likely face, so my current feeling is that even considering I'd have any chance of getting in and contributing to change things was nothing more than a momentary pipe dream.

Anyway, and in regard to the current accessibility infrastructure on macOS, I think that the saying "You can't polish a turd" is a perfect summarization of my thoughts.

I can't make informed comments on macOS development, as it's all done in private by Apple. However, having followed Linux screen reader development in detail over the years, I notice that the bugs that users report are often not in the screen reader at all. Instead, they're in user interface components of the operating system or applications. Screen reader developers have limited ability to work around those bugs. If the accessibility API is used incorrectly, the screen reader may not be able to obtain information that it needs to present the interface appropriately to the user.

Writing another screen reader wouldn't solve problems that lie elsewhere. If most of the issues are in other system or application components, they'll need to be fixed by Apple or by third-party developers.

I'm simply not convinced that writing another screen reader would bring significant benefits, especially in view of the amount of work involved in creating and maintaining it. It would probably be a full-time job for a couple of people, at least.

If you want better accessibility from Apple products, your best option is to pursue improved cooperation with Apple - or focus your accessibility efforts instead on a more open platform.

Here's the thing, people been reporting, myself included the same bugs, long standing mite I add to apple accessibility and still have yet to be fixed. There comes a point where reporting the same bugs over and over after each OS upgrade, in the hopes that said bugs are fixed but not gets tiring. It's why many of us are saying mac OS is going down hill. Honestly, if they treat Mac the same way as they do IOS we wouldn't have as much issues now would we. Correct me if I'm wrong but that just proves my point here. We payed lots of money for these devices, we expect VO and other accessibility products to work just as much efficiently as sighted people be able to use these devices. And don't tell me that apple listens because if they did, they would do a much better job at trying to fix most of these long standing bugs that's been round for years. Safari not responding anyone? I don't doubt that bugs do indeed get fixed but they're little bugs here and there. honestly, if they had one release cycle, not adding features that most of us don't give too f***s about and actually work on bug patching my faith in humanity would be restored but I've yet to see that happen. There comes a point where accountability has got to be taken into account here and we need to stop continuously praising apple to keep their PR and ego up and we need to have a come to Jesus meetin'. When Steve was alive things were much, much better in my honest opinion. Those of you who have been on mac for as long as I have know what I mean. And I grateful for the accessibility we do have across the board? Sure and that's one of the reasons I like apple. But adding accessibility and maintaining accessibility features is apart of the job and they need to be just as much serious about maintaining just as much as adding accessibility. If they don't and they let bugs pile up bug after bug then maintaining becomes more difficult to withstand.

I suspect a big part of the problem is the fact software is released every year. There were almost 2 years between the releases of Tiger and Leopard, about the same for Snow Leopard, and the same for Lion. I used Snow Leopard in 2013, and it was very stable. I didn't really use Lion extensively, so I can't speak to how well VoiceOver worked. Since Mountain Lion, every release has come out roughly once a year, and I think this is where the problems started. If there wasn't this ridiculous push to release new software every year in time for new hardware releases, we'd probably be in better shape. Why not take 2 years between releases to make sure everything is as polished as it can be? Oh well, we'll have to wait for Tim's successor for any hope of that happening, and I'm personally not holding my breath. Steve Jobs was a jerk, which is an understatement of course, but I must abide by the rules of this website. Having said that, at least he got things done.

VoiceOver reports that an application is not responding, but it probably isn't responsible for causing the issue. As in all modern graphical environments, the screen reader uses the accessibility API to retrieve information from the application. If the application doesn't quickly or ever return data in response to the API calls, that's a bug in the application or in its underlying user interface components.

The difference with Windows screen readers is that they don't report the problem - they just become totally unresponsive when similar situations occur. At least VoiceOver tells you about it.

A proper fix may involve redesigning the services that support the accessibility API, or the way in which the API is implemented. I expect someone at Apple has a plan, but they need to make the decision to commit the necessary resources to implementing the right solution - not just fixing individual instances.

Developing a new screen reader is not a task for an individual developer for several reasons. First, it is a huge job and needs to be constantly maintained. This is more than a full time job. Secondly, one has to worry about planning for long term continuation of the project. What happens when the original developer moves on or something happens to them?

Nice as it is for companies to include screen readers with their OS, such screen readers do not seem to be as powerful and useful as third party solutions. After all, the third party solutions only have one focus - Make the screen reader experience be the best possible. Also, third party solutions can release updates as needed and not have to tie their schedule to updates to the overall OS.

Screen readers like JAWS and NVDA also have the advantage of making use of custom scripts and plug-ins to customize how particular applications work. Even if a built in screen reader may make the UI perfectly "accessible" that doesn't necessarily mean that the UI is can be used efficiently or productively by the user. Even the most accessible programs often benefit from customizations and special hotkeys or shortcuts to make it efficient and productive to use with a screen reader.

Anyway, if people are interested in seeing how a real community driven screen reader can be developed and kept going, check out the podcast about NVDA below. With community development and support, this open source screen reader is not only free but offers very comparable functionality to other costly commercial screen readers.

1250 12-12-12 NVDA, A Free Open-Source Screen Reader
Show Notes
Hosts Peter Torpey and Nancy Goodman Torpey interview Mick Curran, one of the co-lead developers of NVDA, a free open-source screen reader for Windows. Learn how the project grew from a do-it-yourself effort to a powerful tool used daily by thousands of visually impaired people around the world and how you can contribute to its success.

--Pete

Ask a sited person who uses safari then or anything else that uses web kit. if people are getting safari not responding when voiceover is on but it does not happen when voiceover is on at all, that's voiceover's fault. Voiceover doesn't play nice with complex pages but you don't see any sited people complaining the same things with safari do you? If they can still move the mouse right after a page is loaded, totally not an app problem. If voiceover still lags and is saying an app is not responding after things are loaded or in Safari's case, after a page is loaded and this happens repeatedly to the point of it freezing, it's yet again, the screen reader's, fault! So you see, no, it's not an application fault what so ever. Again, I've heard from some people that voiceover is only using only, one core so anything outside of that, there can be issues. Again, nothing wrong with VO reporting that an application is not responding, as long as said app is truly not responding. But if it's not responding because voiceover says so, we got a bigger problem. I think I rest my case. Not every screen reader is built the same way and layered the same way. While you may be write with linux, that does not mean that mac does exactly the same thing. While it may go off of apples own accessibility API, interface elements and voiceover, as long as you code them using standard UI kit controls, should work out of the box. I know that's the case with swift. anything custom is the devs responsibility. Could apple's accessibility API play a part here into some of these problems? Maybe but we'll never know. The only real test is to write a basic, and I say basic, screen reader using apple's accessibility API and see if it performs the same, exact, way. Never mind that because I heard docs for that is pretty sparse as far as I know.

The pendulum swings. When it swings back to pretty good, stick with that upgrade. Don't continue upgrading, or you'll be asking for problems. Wait until the pendulum swings back again.

Oh trust me I didn't upgrade. Still on macOS 12 and I plan to keep it that way. Plus this old mac book pro don't go past that, I mean, well, it could ** but kind of don't wanna do so. This my main machine. Logic is, if it ain't broke don't fix it. And my stuff ain't broke. Still able to do a lot of what I could do before.

In response to Jason White, on macOS there are accessibility issues everywhere, and that is part of the problem.

I have no idea what the situation is like on Linux, but I'm way too familiar with the situation on macOS, and will provide some examples to demonstrate them.

On macOS, the accessibility infrastructure is completely orthogonal to the window server, as is also likely the case on Linux, since if I recall correctly it usually works on top of DBus, whereas on macOS it relies on a non-standard multi-producer single-consumer IPC primitive called Mach Ports. The accessibility API is also one of the still actively used remnants of the mostly deprecated and even discontinued Carbon framework, which was originally created to ease the transition from legacy macOS to modern macOS a quarter of a century ago, and was never intended to be the primary way of interacting with the system, but in this case it's actually the only way, which is why Apple can't put it to sleep along with the rest of Carbon. Why Apple chose to implement support for the consumer side of the accessibility infrastructure in Carbon instead of Cocoa when, to my knowledge, legacy macOS did not have an accessibility infrastructure, is a total mystery to me.

The most glaring design problem in that API is that it's blocking. This is not an issue inherited from Mach Ports, as they can be used asynchronously, nor is this a problem in CoreFoundation's support for Mach Ports, as they are fully integrated with its run loops making the asynchronous way the only option, but somehow, and for reasons that I cannot understand, whoever designed this crap decided that the synchronous blocking design was the best for everyone so it should be the default and only available option. The result of this implementation is that, if you set the request timeout for 5 seconds, and 5 system services decide to accept accessibility messages but not actually respond to them, then unless you black list those services, you'll be looking of at least 25 seconds of initialization delay, because of course the accessibility API is also not thread-safe, so you can't just keep making requests to other processes on different threads to mitigate the problem for the user.

Then there are the accessibility observers, which are useful for things like registering to be notified when an element is destroyed, or when its value or title changes. This time they couldn't make it synchronous so they give you a CoreFoundation event source, which is an implementation of CFRunLoopSource that you can add to as many CoreFoundation run loops as you wish to receive the notifications. These implementations are supposed to be thread-safe since invalidating them affects all the run loops on which they are registered, however the ones provided by the accessibility API are not, so if you do invalidate, remove, and release one while it's firing on another thread you end up with a race condition, making this yet another API design flaw.

Still on the subject of accessibility observers, whenever you register to observe some event type, you can pass a pointer to some context data that you can use to refer back to some state related to the event being handled. Given the asynchronous yet not thread-safe nature of accessibility event sources, you can't really expect to be clear to destroy the object pointed at by the aforementioned pointer even after invalidating all the event sources pointing at it, and of course the accessibility API provides no way of knowing when it's definitely safe to do that, so unless you only schedule those event sources on a single run loop, and also schedule their invalidation on the same run loop, your only options are to leak memory or risk a use after free which is an actual memory safety issue, so this is yet another design flaw.

If all the above wasn't bad enough, Apple has also, for some reason that I cannot understand, stopped adding accessibility identifier declarations to the API's headers, so often the only way you end up learning about them is by asking every element to enumerate the ones they implement, and then try to figure out what the ones you don't know actually do and whether they are useful to you. This would be acceptable if we were talking about a private API, but this API is public so I can't even understand why they do this, and from my experience I can tell that the subset of identifiers that are actually declared is a minority.

Although the aforementioned problem with undeclared identifiers has a straight forward solution in the case of passive information that you can query, the same cannot be said for observer notifications. The problem here is that there's no way to ask a process to list all the event types that you can register to be notified about, so you only know what you know, and if you want to learn more the only way is to reverse engineer VoiceOver to sniff on its accessibility API calls and ultimately its Mach Ports. This, in my opinion, is totally unacceptable, especially since Apple includes clauses in their license agreement to void your license if you reverse-engineer their products, which in my case is totally unenforceable since reverse-engineering is explicitly permitted in EU law, but might be legally binding in other jurisdictions.

On the provider side, and beyond the aforementioned problems with undeclared identifiers, while Cocoa does provide some accessibility by default, to my knowledge Apple does not document or even adopt a navigation pattern in their own applications, so as I mentioned previously, it becomes totally impossible to provide a blanket implementation that can tackle and abstract away most navigation problems from users and only have to deal with sporadic outliers, because pretty much everything is an outlier, making the macOS accessibility case a very high maintenance nightmare. Add to this the fact that some glaring problems, like the incorrect implementation of accessibility in the ubiquitous NSTextView that is part of Cocoa and has been like that for at least over a decade now, the theory that accessibility on macOS is nothing more than a totally neglected aspect that they just maintain to tick the accessible product checkbox suddenly becomes very plausibleΩ.

My coworkers did try AXSwift before I got hired, and were having all the issues that I described because AXSwift itself does not address any of them, and likely has never actually been used in a production environment, as it had lots of stability issues due to making incorrect assumptions about the underlying accessibility API. My first task right after joining my current job was precisely to get rid of AXSwift and replace it with the Element module that I wrote for Vosh, which abstracts away all the memory safety and concurrency crap by creating its own async executor and only dealing with anything related to the accessibility calls on a single dedicated background thread.

The problem is that some design issues cannot be abstracted away, so although my module provides an elegant, ergonomic, idiomatic, and safe async Swift interface that addresses all the potential stability issues, at the end of the day its async executor still has to serialize everything on the dedicated background thread due to the aforementioned blocking API, so as I mentioned before, if I do happen to accidentally talk to an application or service that accepts accessibility requests but does not reply back, the dedicated thread is still blocked waiting for a reply and cannot make any other scheduled calls. This is not an issue to the work I'm currently doing, but in a screen-reader where users expect quick feedback it is a very big problem, and can only be worked around with a black list of bundle identifiers of applications and services that are known to misbehave.

As for the inconsistent navigation issues, they cannot be feasibly tackled from the consumer side, as ensuring proper consistency is the job of Cocoa and individual applications themselves. It is possible to tackle the problem to deal with individual Cocoa objects and applications on a case by case basis, however that's an extremely high maintenance burden that I am unwilling to carry alone.

The reason why I'm posting these details is not to say that these problems cannot be addressed, which they obviously can as evidenced by the fact that VoiceOver itself does exist. What I'm trying to convey here is that the whole accessibility infrastructure on macOS is a huge mess with high maintenance requirements that needs to be redone properly from scratch. Reverse-engineering the existing API and reimplementing it properly is a viable option, but since even that would not solve the navigation consistency problems, the motivation to do it is simply not there. Properly solving this problem would require a huge multi-team coordination effort inside Apple that is unlikely to never happen until they decide to remake macOS.

@BeMyEyes, you guys already own Applevis, how about a screenreader for our beloved Macs? 🥺😀
Jokes aside, Since we do pride on Apple's security measures, I don't think Apple would ever allow any 3rd party software to gain such a deep access to system as Voiceover.

This is the job of Apple, they created VO and they need to be held responsible for VO. Who knows if Sara shows up the members of AppleVis ask her if apple ever going to update VO and not just release nice shyning features.

So, I was reading the topic, and I'm wondering something. Would it help to send Apple feedback about the more complex bugs in the accessibility api and other stuff that mostly went over my head? If someone who actually understands the nuts and bolts that make VoiceOver what it is, maybe it might get the deserved attention. Rather than having to work backward from a bug on the consumer end, maybe it could be approached from a more technical viewpoint, if the right people sent that sort of technical feedback.

Perhaps one or more of the advocacy groups that work on technology accessibility issues could take this up with Apple as well. A good discussion of rearchitecting the macOS accessibility infrastructure seems warranted, I think, given some of the details that have been shared in this thread. It's also in Apple's long-term interests to do whatever redesign and reimplementation is needed, to reduce long-term maintenance costs while improving reliability.

First, thank you for the insights from the work you've doen for Vosh, that was very informative.
However, what I've not understood yet, do you mean that many applications respond to Accessibility Requests differently? So that there's no unique pattern? If so, why is the Accessibility infrastructure called API then at all?
Also, does Apple's VoiceOver use the Carbon-era API for consuming Accessibility information? If so, how can they handle all what you've mentioned?
I've always thought (by now) that the UI frameworks like AppKit provide views that we can use and those views implement stuff for posting accessibility notifications (I think they are needed for the AX observers, for example), performing actions (e.g. AXPress) and so on. If this is the case, why is this not the same for every application if they use a button control, for example?
I'm very interested in diving deeper into this topic...

There is a big difference between VO upgrade and fixing bugs. Every year JAWS release major update of it. Every three month they release jaws for bugs. Update is to keep up with iOS updates and apps. Bugs well we know why. We do not bugs to get to the point of NY hotels. Saying we maybe do not know how to use VO is blaming the patient instead of the doctor.

However, what I've not understood yet, do you mean that many applications respond to Accessibility Requests differently? So that there's no unique pattern? If so, why is the Accessibility infrastructure called API then at all?

Also, does Apple's VoiceOver use the Carbon-era API for consuming Accessibility information? If so, how can they handle all what you've mentioned?

What I mean is that there are problems on both ends of the infrastructure. The consumer side, which is where VoiceOver operates, has to deal with an extremely badly design Carbon API, and the producer side, which is where applications use, is highly inconsistent, using a gazillion of different attribute, parameterized attribute, action, and notification identifiers, most of which aren't even declared let alone documented. Add to that some long standing bugs in Cocoa itself, which every graphical application on macOS depends on to draw anything resembling a native UI these days, and finally a highly inconsistent first-party interface language and total nonsense accessibility element hierarchy, and the result is a total mess and very low quality accessibility experience that in my opinion can only be properly fixed by starting over.

Also, does Apple's VoiceOver use the Carbon-era API for consuming Accessibility information? If so, how can they handle all what you've mentioned?

If you read my posts, you will notice that I mentioned workarounds for all the problems, however none of them are proper workarounds, which is why people experience so many problems with VoiceOver on macOS. If you have a major problem, and your solution is consistently to go after its symptoms instead of tackling the root cause, eventually you end up with an extremely complex mess that people grow increasingly weary of actually touching, because bugs have this tendency to interact with other bugs in unpredictable ways, so the longer you postpone a proper solution, the higher the chance of a change to plug a specific symptom ending up creating other problems in completely unrelated and unexpected places. In the software engineering circles, we call this technical debt, and I personally go out of my way to never contribute to it.

I've always thought (by now) that the UI frameworks like AppKit provide views that we can use and those views implement stuff for posting accessibility notifications (I think they are needed for the AX observers, for example), performing actions (e.g. AXPress) and so on. If this is the case, why is this not the same for every application if they use a button control, for example?

You seem to be confusing passive attributes and user actions with notifications. Performing an action on an element, like pressing a button, does not trigger an accessibility notification. In these cases you use the blocking API to send a message through the accessibility API and hopefully the application on the other end replies back so your thread gets unblocked and you can read the result of the requested operation. At the lowest level this is implemented using Mach Ports as I mentioned, and the convention is that the side that sends the message also provides a Mach Port for the recipient to reply back. Accessibility notifications are completely different from this in that they are spontaneous events that any observed element can send at any time, however they also work on top of Mach Ports.

As for the identifiers, while some of them are declared in headers bundled with Xcode, most of them are not, and instead of trying to stick to a fixed set, Apple keeps adding more and more of them to first-party applications and frameworks, and updating VoiceOver to take advantage of them but without documenting or even declaring them anywhere. While most of the identifiers used by Cocoa itself seem to be properly declared, none of the ones added by other frameworks are. WebKit is likely one of the biggest offenders here, as pretty much every HTML element implements at least one undocumented and undeclared identifier, but even Blink, which is Chrome's web engine, adds its own special identifiers.

Regarding notification identifiers, Terminal and Apple Pay are two examples that I've personally witnessed sending undeclared notifications to VoiceOver, and this was only possible to accomplish through reverse-engineering, because you do not receive notifications that you don't ask for, and there's no way to list all the notifications implemented by an element. Part of why I find reverse-engineering so annoying in this case is because I'm messing with a service that I depend on, making it a task that requires a lot of creativity to accomplish.

One of the things that I didn't even figure out at the time, probably because I never actually spent time conducting a proper investigation, but it's definitely not something obvious, is how VoiceOver actually gathers information about custom application rotors. The accessibility API does provide a way to list actions with human-readable descriptions and activate the action rotor, but I have absolutely no idea of how all the other rotors are accessed.

It is difficult to keep tacking fixes onto legacy code that has been around for a long time. Nice as it would be to rewrite code from scratch taking new paradigms into account, that can be a major task sucking up a lot of resources. Plus, as others have pointed out, then one has to worry about backwards compatibility.

A lot has changed in how operating systems and computers work over the past few decades. Just taking web accessibility into account is very different now than it used to be since many web pages are more like apps these days than static text.

That being said, even JAWS for Windows went through some major changes under the hood in the past few years because of changes udner the hood that happened in the OS. The OSM or Off Screen Model upon which JAWS was developed just couldn't cut it any more. Thus JAWS had to migrate to a completely different way of accessing text displayed on the screen and working with accessibility elements. Even Microsoft Windows has been changing from MSAA to UIA and whatever the next technologies are for trying to enhance accessibility as newer operating systems and programs take advantage of new technologies and ways of doing things.

All this to say that there are very real trade offs in staying on the legacy code band wagon or ditching everything and starting from scratch. It can be a big job, and if most things work, it is often easier to keep them working and patch problems.

--Pete