The State of Screen readers in macOS

This website already has a bug tracker, though I don't know how comprehensive it is. I would hope Apple looks at it, but you never know. Someone claiming to be from Apple came on here a while ago and said employees aren't allowed to look at the website during work hours. If true, this is ridiculous and utterly counterproductive! At best, it's someone trolling, but at worst, it demonstrates the problems with Apple's culture. I'm sure the people working on these features want to do all kinds of wonderful things, but can't and/or unwillingly have their priorities shifted because the corporate overlords have their hands tied.

As for creating a screen reader, my understanding is that Apple doesn't document all the APIs and other things required to create a fully functional one. It should technically be possible on macOS, but not any other Apple platform because they're so tightly controlled.

When was the last time Apple did a real update of VO? Yes, they added nice features and fix bugs when they remember to do so. With windows NVDA and JAWS update their software to address bugs and make it more easy to use. Before you jump on me, do like VO but just asking. Do not use MAC but heard stories about it from here.

I think we need a third-party screen reader. And I would pay for it if it was made available. Voiceover is fine in a lot of ways. And I think an additional option could be beneficial.

VO use to be great! Member snow leopard anyone? It started going down hill when Tim took the job and even shown more bugs in later OS upgrades. There are bugs that have existed for a long time that still, has yet to get fixed. My honest opinion and I've been saying this for a long time, VO needs, a complete redesign and code rewrite! I don't care, apple has all the resources to do just, exactly that. Apple don't fix a lot of these bugs and in some cases, gotten worse. Sure, some have been fixed but VO has more bugs than been fixed Lol. Plus I've heard that VO runs on one core only. I don't know if that's still true but look at the times. It's clearly time for an over hall. However, I wouldn't mind a third party screen reader if it came to be, fact I'd welcome it.

I agree with the last poster. I am proud to be stuck in tutorial hell for the last 10 years so my dev skills....... But I do have experience with lots of different screen readers on different oses and voiceover on mac is flawed by design in many areas now. But until this moment... The best we can do is still to report as much as we can, and also acknowledge that it's very exhausting so nobody should feel guilty of not doing enough or not doing at all...

I'm not familiar of how NVDA actually got started since when I went blind all accessibility technology was already quite advanced. However I did begin work on a screen-reader project in the past, and hopefully attracting other developers to help with it was the reason why I ended up creating an account here after lurking for years. Unfortunately despite all the enthusiasm that my initial demo caused at the time, I did not get that much in terms of contributions, and ultimately ended up giving up on that project because I was spending more time reverse engineering macOS, which is something that I can do just fine but don't consider quality time, than progressing with the project. Fortunately my work on that project actually ended up landing me a job last year so not everything was bad.

There are many problems with the accessibility infrastructure on macOS, some resulting from poor design and some resulting from neglect, but overall the consumer side of the accessibility infrastructure is extremely quirky and full of pitfalls, requiring very high maintenance with lots of testing as from my hands-on experience, there's absolutely no way to approach a large number of cases with a blanket implementation to only have to deal with the exceptions because pretty much everything is an exception. Even things as simple as properly announcing and responding to accessibility requests are not implemented properly in some system services, making it unfeasible to casually talk to any process that have an open connection to the window server without maintaining a blacklist of services that do not behave properly, as the way the accessibility API works makes it impossible to avoid the resulting negative side effects.

Last Saturday, while checking out some gear from the on-line Apple Store, I casually followed a link to a page where people can apply to work at Apple, and for the first time n my life I actually gave it a serious thought, because as a Mac power user who absolutely depends on accessibility to use computers, and as a developer with a lot of easily provable experience with the accessibility infrastructure on macOS among many other fields in software engineering, a perfectly clean criminal and financial record, and a long relationship with Apple that has always been in good standing, I believe that I could actually provide value by helping turn this ship around as an insider. However I am already invested in other options, like creating my own company, and am still working for the people who hired me last year, plus even if somehow I got through Apple's hiring process, that would still require relocating to a different country, and I have absolutely no idea of how much corporate management friction I would likely face, so my current feeling is that even considering I'd have any chance of getting in and contributing to change things was nothing more than a momentary pipe dream.

Anyway, and in regard to the current accessibility infrastructure on macOS, I think that the saying "You can't polish a turd" is a perfect summarization of my thoughts.

I can't make informed comments on macOS development, as it's all done in private by Apple. However, having followed Linux screen reader development in detail over the years, I notice that the bugs that users report are often not in the screen reader at all. Instead, they're in user interface components of the operating system or applications. Screen reader developers have limited ability to work around those bugs. If the accessibility API is used incorrectly, the screen reader may not be able to obtain information that it needs to present the interface appropriately to the user.

Writing another screen reader wouldn't solve problems that lie elsewhere. If most of the issues are in other system or application components, they'll need to be fixed by Apple or by third-party developers.

I'm simply not convinced that writing another screen reader would bring significant benefits, especially in view of the amount of work involved in creating and maintaining it. It would probably be a full-time job for a couple of people, at least.

If you want better accessibility from Apple products, your best option is to pursue improved cooperation with Apple - or focus your accessibility efforts instead on a more open platform.

Here's the thing, people been reporting, myself included the same bugs, long standing mite I add to apple accessibility and still have yet to be fixed. There comes a point where reporting the same bugs over and over after each OS upgrade, in the hopes that said bugs are fixed but not gets tiring. It's why many of us are saying mac OS is going down hill. Honestly, if they treat Mac the same way as they do IOS we wouldn't have as much issues now would we. Correct me if I'm wrong but that just proves my point here. We payed lots of money for these devices, we expect VO and other accessibility products to work just as much efficiently as sighted people be able to use these devices. And don't tell me that apple listens because if they did, they would do a much better job at trying to fix most of these long standing bugs that's been round for years. Safari not responding anyone? I don't doubt that bugs do indeed get fixed but they're little bugs here and there. honestly, if they had one release cycle, not adding features that most of us don't give too f***s about and actually work on bug patching my faith in humanity would be restored but I've yet to see that happen. There comes a point where accountability has got to be taken into account here and we need to stop continuously praising apple to keep their PR and ego up and we need to have a come to Jesus meetin'. When Steve was alive things were much, much better in my honest opinion. Those of you who have been on mac for as long as I have know what I mean. And I grateful for the accessibility we do have across the board? Sure and that's one of the reasons I like apple. But adding accessibility and maintaining accessibility features is apart of the job and they need to be just as much serious about maintaining just as much as adding accessibility. If they don't and they let bugs pile up bug after bug then maintaining becomes more difficult to withstand.

I suspect a big part of the problem is the fact software is released every year. There were almost 2 years between the releases of Tiger and Leopard, about the same for Snow Leopard, and the same for Lion. I used Snow Leopard in 2013, and it was very stable. I didn't really use Lion extensively, so I can't speak to how well VoiceOver worked. Since Mountain Lion, every release has come out roughly once a year, and I think this is where the problems started. If there wasn't this ridiculous push to release new software every year in time for new hardware releases, we'd probably be in better shape. Why not take 2 years between releases to make sure everything is as polished as it can be? Oh well, we'll have to wait for Tim's successor for any hope of that happening, and I'm personally not holding my breath. Steve Jobs was a jerk, which is an understatement of course, but I must abide by the rules of this website. Having said that, at least he got things done.

VoiceOver reports that an application is not responding, but it probably isn't responsible for causing the issue. As in all modern graphical environments, the screen reader uses the accessibility API to retrieve information from the application. If the application doesn't quickly or ever return data in response to the API calls, that's a bug in the application or in its underlying user interface components.

The difference with Windows screen readers is that they don't report the problem - they just become totally unresponsive when similar situations occur. At least VoiceOver tells you about it.

A proper fix may involve redesigning the services that support the accessibility API, or the way in which the API is implemented. I expect someone at Apple has a plan, but they need to make the decision to commit the necessary resources to implementing the right solution - not just fixing individual instances.

Developing a new screen reader is not a task for an individual developer for several reasons. First, it is a huge job and needs to be constantly maintained. This is more than a full time job. Secondly, one has to worry about planning for long term continuation of the project. What happens when the original developer moves on or something happens to them?

Nice as it is for companies to include screen readers with their OS, such screen readers do not seem to be as powerful and useful as third party solutions. After all, the third party solutions only have one focus - Make the screen reader experience be the best possible. Also, third party solutions can release updates as needed and not have to tie their schedule to updates to the overall OS.

Screen readers like JAWS and NVDA also have the advantage of making use of custom scripts and plug-ins to customize how particular applications work. Even if a built in screen reader may make the UI perfectly "accessible" that doesn't necessarily mean that the UI is can be used efficiently or productively by the user. Even the most accessible programs often benefit from customizations and special hotkeys or shortcuts to make it efficient and productive to use with a screen reader.

Anyway, if people are interested in seeing how a real community driven screen reader can be developed and kept going, check out the podcast about NVDA below. With community development and support, this open source screen reader is not only free but offers very comparable functionality to other costly commercial screen readers.

1250 12-12-12 NVDA, A Free Open-Source Screen Reader
Show Notes
Hosts Peter Torpey and Nancy Goodman Torpey interview Mick Curran, one of the co-lead developers of NVDA, a free open-source screen reader for Windows. Learn how the project grew from a do-it-yourself effort to a powerful tool used daily by thousands of visually impaired people around the world and how you can contribute to its success.

--Pete

Ask a sited person who uses safari then or anything else that uses web kit. if people are getting safari not responding when voiceover is on but it does not happen when voiceover is on at all, that's voiceover's fault. Voiceover doesn't play nice with complex pages but you don't see any sited people complaining the same things with safari do you? If they can still move the mouse right after a page is loaded, totally not an app problem. If voiceover still lags and is saying an app is not responding after things are loaded or in Safari's case, after a page is loaded and this happens repeatedly to the point of it freezing, it's yet again, the screen reader's, fault! So you see, no, it's not an application fault what so ever. Again, I've heard from some people that voiceover is only using only, one core so anything outside of that, there can be issues. Again, nothing wrong with VO reporting that an application is not responding, as long as said app is truly not responding. But if it's not responding because voiceover says so, we got a bigger problem. I think I rest my case. Not every screen reader is built the same way and layered the same way. While you may be write with linux, that does not mean that mac does exactly the same thing. While it may go off of apples own accessibility API, interface elements and voiceover, as long as you code them using standard UI kit controls, should work out of the box. I know that's the case with swift. anything custom is the devs responsibility. Could apple's accessibility API play a part here into some of these problems? Maybe but we'll never know. The only real test is to write a basic, and I say basic, screen reader using apple's accessibility API and see if it performs the same, exact, way. Never mind that because I heard docs for that is pretty sparse as far as I know.

The pendulum swings. When it swings back to pretty good, stick with that upgrade. Don't continue upgrading, or you'll be asking for problems. Wait until the pendulum swings back again.

Oh trust me I didn't upgrade. Still on macOS 12 and I plan to keep it that way. Plus this old mac book pro don't go past that, I mean, well, it could ** but kind of don't wanna do so. This my main machine. Logic is, if it ain't broke don't fix it. And my stuff ain't broke. Still able to do a lot of what I could do before.

In response to Jason White, on macOS there are accessibility issues everywhere, and that is part of the problem.

I have no idea what the situation is like on Linux, but I'm way too familiar with the situation on macOS, and will provide some examples to demonstrate them.

On macOS, the accessibility infrastructure is completely orthogonal to the window server, as is also likely the case on Linux, since if I recall correctly it usually works on top of DBus, whereas on macOS it relies on a non-standard multi-producer single-consumer IPC primitive called Mach Ports. The accessibility API is also one of the still actively used remnants of the mostly deprecated and even discontinued Carbon framework, which was originally created to ease the transition from legacy macOS to modern macOS a quarter of a century ago, and was never intended to be the primary way of interacting with the system, but in this case it's actually the only way, which is why Apple can't put it to sleep along with the rest of Carbon. Why Apple chose to implement support for the consumer side of the accessibility infrastructure in Carbon instead of Cocoa when, to my knowledge, legacy macOS did not have an accessibility infrastructure, is a total mystery to me.

The most glaring design problem in that API is that it's blocking. This is not an issue inherited from Mach Ports, as they can be used asynchronously, nor is this a problem in CoreFoundation's support for Mach Ports, as they are fully integrated with its run loops making the asynchronous way the only option, but somehow, and for reasons that I cannot understand, whoever designed this crap decided that the synchronous blocking design was the best for everyone so it should be the default and only available option. The result of this implementation is that, if you set the request timeout for 5 seconds, and 5 system services decide to accept accessibility messages but not actually respond to them, then unless you black list those services, you'll be looking of at least 25 seconds of initialization delay, because of course the accessibility API is also not thread-safe, so you can't just keep making requests to other processes on different threads to mitigate the problem for the user.

Then there are the accessibility observers, which are useful for things like registering to be notified when an element is destroyed, or when its value or title changes. This time they couldn't make it synchronous so they give you a CoreFoundation event source, which is an implementation of CFRunLoopSource that you can add to as many CoreFoundation run loops as you wish to receive the notifications. These implementations are supposed to be thread-safe since invalidating them affects all the run loops on which they are registered, however the ones provided by the accessibility API are not, so if you do invalidate, remove, and release one while it's firing on another thread you end up with a race condition, making this yet another API design flaw.

Still on the subject of accessibility observers, whenever you register to observe some event type, you can pass a pointer to some context data that you can use to refer back to some state related to the event being handled. Given the asynchronous yet not thread-safe nature of accessibility event sources, you can't really expect to be clear to destroy the object pointed at by the aforementioned pointer even after invalidating all the event sources pointing at it, and of course the accessibility API provides no way of knowing when it's definitely safe to do that, so unless you only schedule those event sources on a single run loop, and also schedule their invalidation on the same run loop, your only options are to leak memory or risk a use after free which is an actual memory safety issue, so this is yet another design flaw.

If all the above wasn't bad enough, Apple has also, for some reason that I cannot understand, stopped adding accessibility identifier declarations to the API's headers, so often the only way you end up learning about them is by asking every element to enumerate the ones they implement, and then try to figure out what the ones you don't know actually do and whether they are useful to you. This would be acceptable if we were talking about a private API, but this API is public so I can't even understand why they do this, and from my experience I can tell that the subset of identifiers that are actually declared is a minority.

Although the aforementioned problem with undeclared identifiers has a straight forward solution in the case of passive information that you can query, the same cannot be said for observer notifications. The problem here is that there's no way to ask a process to list all the event types that you can register to be notified about, so you only know what you know, and if you want to learn more the only way is to reverse engineer VoiceOver to sniff on its accessibility API calls and ultimately its Mach Ports. This, in my opinion, is totally unacceptable, especially since Apple includes clauses in their license agreement to void your license if you reverse-engineer their products, which in my case is totally unenforceable since reverse-engineering is explicitly permitted in EU law, but might be legally binding in other jurisdictions.

On the provider side, and beyond the aforementioned problems with undeclared identifiers, while Cocoa does provide some accessibility by default, to my knowledge Apple does not document or even adopt a navigation pattern in their own applications, so as I mentioned previously, it becomes totally impossible to provide a blanket implementation that can tackle and abstract away most navigation problems from users and only have to deal with sporadic outliers, because pretty much everything is an outlier, making the macOS accessibility case a very high maintenance nightmare. Add to this the fact that some glaring problems, like the incorrect implementation of accessibility in the ubiquitous NSTextView that is part of Cocoa and has been like that for at least over a decade now, the theory that accessibility on macOS is nothing more than a totally neglected aspect that they just maintain to tick the accessible product checkbox suddenly becomes very plausibleΩ.

My coworkers did try AXSwift before I got hired, and were having all the issues that I described because AXSwift itself does not address any of them, and likely has never actually been used in a production environment, as it had lots of stability issues due to making incorrect assumptions about the underlying accessibility API. My first task right after joining my current job was precisely to get rid of AXSwift and replace it with the Element module that I wrote for Vosh, which abstracts away all the memory safety and concurrency crap by creating its own async executor and only dealing with anything related to the accessibility calls on a single dedicated background thread.

The problem is that some design issues cannot be abstracted away, so although my module provides an elegant, ergonomic, idiomatic, and safe async Swift interface that addresses all the potential stability issues, at the end of the day its async executor still has to serialize everything on the dedicated background thread due to the aforementioned blocking API, so as I mentioned before, if I do happen to accidentally talk to an application or service that accepts accessibility requests but does not reply back, the dedicated thread is still blocked waiting for a reply and cannot make any other scheduled calls. This is not an issue to the work I'm currently doing, but in a screen-reader where users expect quick feedback it is a very big problem, and can only be worked around with a black list of bundle identifiers of applications and services that are known to misbehave.

As for the inconsistent navigation issues, they cannot be feasibly tackled from the consumer side, as ensuring proper consistency is the job of Cocoa and individual applications themselves. It is possible to tackle the problem to deal with individual Cocoa objects and applications on a case by case basis, however that's an extremely high maintenance burden that I am unwilling to carry alone.

The reason why I'm posting these details is not to say that these problems cannot be addressed, which they obviously can as evidenced by the fact that VoiceOver itself does exist. What I'm trying to convey here is that the whole accessibility infrastructure on macOS is a huge mess with high maintenance requirements that needs to be redone properly from scratch. Reverse-engineering the existing API and reimplementing it properly is a viable option, but since even that would not solve the navigation consistency problems, the motivation to do it is simply not there. Properly solving this problem would require a huge multi-team coordination effort inside Apple that is unlikely to never happen until they decide to remake macOS.

@BeMyEyes, you guys already own Applevis, how about a screenreader for our beloved Macs? 🥺😀
Jokes aside, Since we do pride on Apple's security measures, I don't think Apple would ever allow any 3rd party software to gain such a deep access to system as Voiceover.

This is the job of Apple, they created VO and they need to be held responsible for VO. Who knows if Sara shows up the members of AppleVis ask her if apple ever going to update VO and not just release nice shyning features.

So, I was reading the topic, and I'm wondering something. Would it help to send Apple feedback about the more complex bugs in the accessibility api and other stuff that mostly went over my head? If someone who actually understands the nuts and bolts that make VoiceOver what it is, maybe it might get the deserved attention. Rather than having to work backward from a bug on the consumer end, maybe it could be approached from a more technical viewpoint, if the right people sent that sort of technical feedback.

Perhaps one or more of the advocacy groups that work on technology accessibility issues could take this up with Apple as well. A good discussion of rearchitecting the macOS accessibility infrastructure seems warranted, I think, given some of the details that have been shared in this thread. It's also in Apple's long-term interests to do whatever redesign and reimplementation is needed, to reduce long-term maintenance costs while improving reliability.

First, thank you for the insights from the work you've doen for Vosh, that was very informative.
However, what I've not understood yet, do you mean that many applications respond to Accessibility Requests differently? So that there's no unique pattern? If so, why is the Accessibility infrastructure called API then at all?
Also, does Apple's VoiceOver use the Carbon-era API for consuming Accessibility information? If so, how can they handle all what you've mentioned?
I've always thought (by now) that the UI frameworks like AppKit provide views that we can use and those views implement stuff for posting accessibility notifications (I think they are needed for the AX observers, for example), performing actions (e.g. AXPress) and so on. If this is the case, why is this not the same for every application if they use a button control, for example?
I'm very interested in diving deeper into this topic...

There is a big difference between VO upgrade and fixing bugs. Every year JAWS release major update of it. Every three month they release jaws for bugs. Update is to keep up with iOS updates and apps. Bugs well we know why. We do not bugs to get to the point of NY hotels. Saying we maybe do not know how to use VO is blaming the patient instead of the doctor.

However, what I've not understood yet, do you mean that many applications respond to Accessibility Requests differently? So that there's no unique pattern? If so, why is the Accessibility infrastructure called API then at all?

Also, does Apple's VoiceOver use the Carbon-era API for consuming Accessibility information? If so, how can they handle all what you've mentioned?

What I mean is that there are problems on both ends of the infrastructure. The consumer side, which is where VoiceOver operates, has to deal with an extremely badly design Carbon API, and the producer side, which is where applications use, is highly inconsistent, using a gazillion of different attribute, parameterized attribute, action, and notification identifiers, most of which aren't even declared let alone documented. Add to that some long standing bugs in Cocoa itself, which every graphical application on macOS depends on to draw anything resembling a native UI these days, and finally a highly inconsistent first-party interface language and total nonsense accessibility element hierarchy, and the result is a total mess and very low quality accessibility experience that in my opinion can only be properly fixed by starting over.

Also, does Apple's VoiceOver use the Carbon-era API for consuming Accessibility information? If so, how can they handle all what you've mentioned?

If you read my posts, you will notice that I mentioned workarounds for all the problems, however none of them are proper workarounds, which is why people experience so many problems with VoiceOver on macOS. If you have a major problem, and your solution is consistently to go after its symptoms instead of tackling the root cause, eventually you end up with an extremely complex mess that people grow increasingly weary of actually touching, because bugs have this tendency to interact with other bugs in unpredictable ways, so the longer you postpone a proper solution, the higher the chance of a change to plug a specific symptom ending up creating other problems in completely unrelated and unexpected places. In the software engineering circles, we call this technical debt, and I personally go out of my way to never contribute to it.

I've always thought (by now) that the UI frameworks like AppKit provide views that we can use and those views implement stuff for posting accessibility notifications (I think they are needed for the AX observers, for example), performing actions (e.g. AXPress) and so on. If this is the case, why is this not the same for every application if they use a button control, for example?

You seem to be confusing passive attributes and user actions with notifications. Performing an action on an element, like pressing a button, does not trigger an accessibility notification. In these cases you use the blocking API to send a message through the accessibility API and hopefully the application on the other end replies back so your thread gets unblocked and you can read the result of the requested operation. At the lowest level this is implemented using Mach Ports as I mentioned, and the convention is that the side that sends the message also provides a Mach Port for the recipient to reply back. Accessibility notifications are completely different from this in that they are spontaneous events that any observed element can send at any time, however they also work on top of Mach Ports.

As for the identifiers, while some of them are declared in headers bundled with Xcode, most of them are not, and instead of trying to stick to a fixed set, Apple keeps adding more and more of them to first-party applications and frameworks, and updating VoiceOver to take advantage of them but without documenting or even declaring them anywhere. While most of the identifiers used by Cocoa itself seem to be properly declared, none of the ones added by other frameworks are. WebKit is likely one of the biggest offenders here, as pretty much every HTML element implements at least one undocumented and undeclared identifier, but even Blink, which is Chrome's web engine, adds its own special identifiers.

Regarding notification identifiers, Terminal and Apple Pay are two examples that I've personally witnessed sending undeclared notifications to VoiceOver, and this was only possible to accomplish through reverse-engineering, because you do not receive notifications that you don't ask for, and there's no way to list all the notifications implemented by an element. Part of why I find reverse-engineering so annoying in this case is because I'm messing with a service that I depend on, making it a task that requires a lot of creativity to accomplish.

One of the things that I didn't even figure out at the time, probably because I never actually spent time conducting a proper investigation, but it's definitely not something obvious, is how VoiceOver actually gathers information about custom application rotors. The accessibility API does provide a way to list actions with human-readable descriptions and activate the action rotor, but I have absolutely no idea of how all the other rotors are accessed.

It is difficult to keep tacking fixes onto legacy code that has been around for a long time. Nice as it would be to rewrite code from scratch taking new paradigms into account, that can be a major task sucking up a lot of resources. Plus, as others have pointed out, then one has to worry about backwards compatibility.

A lot has changed in how operating systems and computers work over the past few decades. Just taking web accessibility into account is very different now than it used to be since many web pages are more like apps these days than static text.

That being said, even JAWS for Windows went through some major changes under the hood in the past few years because of changes udner the hood that happened in the OS. The OSM or Off Screen Model upon which JAWS was developed just couldn't cut it any more. Thus JAWS had to migrate to a completely different way of accessing text displayed on the screen and working with accessibility elements. Even Microsoft Windows has been changing from MSAA to UIA and whatever the next technologies are for trying to enhance accessibility as newer operating systems and programs take advantage of new technologies and ways of doing things.

All this to say that there are very real trade offs in staying on the legacy code band wagon or ditching everything and starting from scratch. It can be a big job, and if most things work, it is often easier to keep them working and patch problems.

--Pete

It is difficult to keep tacking fixes onto legacy code that has been around for a long time. Nice as it would be to rewrite code from scratch taking new paradigms into account, that can be a major task sucking up a lot of resources. Plus, as others have pointed out, then one has to worry about backwards compatibility.

Yeah but strangely, Apple themselves don't seem to give two craps about backwards compatibility when it comes to third-party developers, as Most of Carbon was discontinued in Catalina when Apple decided to drop support for 32.bit applications, plus they almost drove Rogue Amoeba out of business by making it impossible to do proper audio routing using a third-party application in macOS Big Sur without asking the user to disable system protections. However the rules don't seem to apply to their own stuff apparently, so the parts of Carbon required to make Apple code work are exempted from deprecation and discontinuation for as long as Apple themselves depend on them. Also we're talking about an API that is very rarely used by third party applications, so updating it would affect very few developers outside Apple.

The more I actually think about this, the more ridiculous the situation feels, because Steve Jobs himself berated Adobe for dragging their feet when it came to supporting Cocoa on macOS as part of his famous rant on flash in 2010, as I will quote below:

Flash is a cross platform development tool. It is not Adobe’s goal to help developers write the best iPhone, iPod and iPad apps. It is their goal to help developers write cross platform apps. And Adobe has been painfully slow to adopt enhancements to Apple’s platforms. For example, although Mac OS X has been shipping for almost 10 years now, Adobe just adopted it fully (Cocoa) two weeks ago when they shipped CS5. Adobe was the last major third party developer to fully adopt Mac OS X.

Lastly I feel the need to clarify that my thoughts both now and back then were and are that Steve Jobs was and is absolutely right on this subject. My criticism is about modern Apple not living up to their own standards and legacy by dragging their own feet when it comes to delivering quality, not their stance on Flash which I'm only referring to here to highlight the difference.

This is probably the most civilized and informative thread I've ever seen on applevis. With my limited dev experience but very good experience now with reporting front end bugs to apple and understanding the hundreds of exception to each rule of how VO works (or doesn't) on macos, all this seems just so right to me, as just after three weeks I knew that VO had serious design flaws way beyond just the surface sscreen reader. Anyways. Thanks for sharing your experience with the community, I am sure that just the feeling that no we are not wrong and there are indeed real problems and some clever people know them better than us is invaluable to many, including myself.

Thank you for the clarification. But what I‘ve still not get completely by now is why responding to Accessibility Requests is not unique throughout one framework.
Let‘s say I have an AppKit-based appplication running that only has one button. I send a request to get the focused element. The application responds back with some stuff. Isn‘t exactly that response then already implemented because the native controls implenet the NSAccessibility protocol?
Or is it such that each view responds completely different, if I understood it right, with different accessibility identifiers that are not documented at all? So that the NSAccessibility protocl is implemented differently on each view?
And another question, does all this (Sending requests / receiving responses) happen via an Abstraction Layer (e.g. via AXVisualSupportAgent or another AX service) that handles the low-level Mach IPC? Or does this work differently?
And is there a way to check out how some views have actually implemented how to respond to accessibility requests? I’m very interested in this topic.

Thank you for the clarification. But what I‘ve still not get completely by now is why responding to Accessibility Requests is not unique throughout one framework.

It is, and I don't understand how anything I said made you think differently.

Let‘s say I have an AppKit-based appplication running that only has one button. I send a request to get the focused element. The application responds back with some stuff. Isn‘t exactly that response then already implemented because the native controls implenet the NSAccessibility protocol?

Yes...

Or is it such that each view responds completely different, if I understood it right, with different accessibility identifiers that are not documented at all? So that the NSAccessibility protocl is implemented differently on each view?

Well there are the common ones, which are implemented by Cocoa and are mostly declared in C header files shipped with Xcode, and then there are a bunch of custom ones that are not documented anywhere. For example AXLink is an example of a custom attribute identifier used in WebKit that is not actually declared in any public header and is quite useful. Another very useful, quite ubiquitous, and totally undeclared identifier is AXChildrenInNavigationOrder, which contains exactly the same information as AXChildren except the former is sorted, so they could have just replaced the latter instead of creating a completely new undocumented and undeclared identifier. Among the declared identifiers there's also the inconsistent usage of AXTitle, AXValue, AXValueDescription, AXText, and even indirect ones like AXTitleElement, to provide human-readable information for the screen-reader to announce.

The above examples are all of accessibility attributes that can be listed and their values easily queried, so figuring out how to use them is trivial, with the only real problem being their inconsistent use even in Cocoa. Parameterized attributes are a bit harder to figure out since in addition to their identifiers they also require a parameter of a specific data type, so when they are not documented you have to guess the data type that they accept, which is often but not always trivial to infer. Actions are expected to not be declared or documented since applications are encouraged to add custom ones which appear in the action rotor, and you have the option of also listing their human-readable descriptions. Custom rotor information seems to be hidden somewhere. Notifications are a problem since you can't actually ask an element to tell you which ones you can subscribe to as you only receive notifications that you have already subscribed.

Beyond the above there's the inconsistent element hierarchy, where in many cases you end up with elements that are supposed to be focusable or actionable but the element that gets the focus or action is elsewhere in the element hierarchy, the deep accessibility trees where most nodes have absolutely no relevant information but are still marked as relevant by their frameworks, and plenty of other forms of completely unnecessary complexity making the accessibility structure totally unpredictable even in first-party applications.

And another question, does all this (Sending requests / receiving responses) happen via an Abstraction Layer (e.g. via AXVisualSupportAgent or another AX service) that handles the low-level Mach IPC? Or does this work differently?

At the lowest level it's direct Mach Port communication between accessibility consumers like screen-reader processes and accessibility producers like system application processes, with the abstraction being provided by the aforementioned accessibility API in the Carbon framework. There's probably some authentication done against Transparency, Consent, and Control at some point so that the system can tell whether you've been granted the right privileges, but investigating this was outside the scope of my interest. AXVisualSupportAgent never actually appeared anywhere relevant to me, and from the name I guess it might have something to do with tasks that require interacting with the window server, like rendering the frame to convey the screen-reader focus on accessibility elements, magnifying the contents of the focused element, providing screen-recognition, and probably even rendering the opaque black view for screen curtain, but this is just a purely speculative and uninformed guess.

And is there a way to check out how some views have actually implemented how to respond to accessibility requests? I’m very interested in this topic.

Nothing short of experimentation and reverse-engineering that I can think of. Apple provides a tool called Accessibility Inspector that I've never actually used myself , but from what I gather, that tool does not provide any information beyond what you can already obtain using the accessibility API. If you build an app with any of those views you can always traverse the view hierarchy to read all the information related to the NSAccessibilityProtocol, and you can also use the Objective-C runtime to inspect the dynamic implementation of all the relevant Cocoa objects.

As for reverse-engineering, my workflow is just installing macOS on a virtual machine, disabling all the security protections on that installation, installing Xcode's command-line tools inside, and then using a combination of linker and debugging hacks to inject code and extract information out of a running VoiceOver instance. Since I depend on VoiceOver to actually use the virtual machine myself, my interactions with the debugger are always performed by connecting to the virtual machine through ssh from the host system, and then switching in and out of its window to produce events that I can either log or trap to observe in the debugger. Being able to write and understand C and assembly code, mastering the debugger so you can write custom scripts for it, and understanding how the dynamic linker works are all very important skills to have in order to even have a chance at succeeding at this kind of task.

Thanks for the clarification. The thing that got me thinking about the provided accessibility information not to be unique in AppKit, for example, is that you've said that some system services do not respond properly to AX requests.
From what I've heard so far, it seems that Apple has never deeply thought on how to actually design the AX Infrastructure, it reminds me of projects that I began on a weekend and that I simply began coding on without thinking of how to really design them.

It's been really clear to me pretty much since I started using VoiceOver on the Mac that there were fundamental issues that are never being addressed. I don't understand all of the above but it's interesting to get some insight into what this means technically.

If you look at the new things Apple adds to VoiceOver, it seems to be mainly quick wins. For example the new Commands editor in VO Utility with Sequoia. It was probably a relatively simple thing to add but it does nothing to change the overall VO experience. Everything they do just feels like extra baggage.

The descriptions above remind me a little of the whole Netscape vs Internet Explorer thing. Maybe this is just me showing my age. But when I was a web developer I remember the pain of having to fight both of these browsers, and in particular Netscape was such a horrible buggy mess and it just got worse and worse. Both companies were just flinging in new stuff trying to outdo each other, but the end result was a less and less stable browser and the web suffered greatly as a result.

Netscape seemed to realise this and so Firefox was born. This was a new browser that seemed to care about web standards, had none of the stability issues and was actually pretty good. But it took so long for it to appear that by that stage it was almost too late already. It has never been able to gain the same level of market domination as Netscape. Maybe that was to do with Microsoft pushing Internet Explorer so hard in Windows, or maybe it was just the time it took to develop.

Now we have Apple competing against Microsoft on one side, and with Google on the other. There is this pressure to release a big new thing every year packed with new features. But this just means that the problem is getting worse every year. I almost see the new VoiceOver features as little apologies - sorry we couldn't actually make it any good but here's something that is shiny to distract you.

So how does this ever get fixed? Or is it just going to get worse until it is unusable and then we all move elsewhere?

There seems to be such a big disconnect between Apple and the community so I don't feel optimistic that they are necessarily going to listen if we say "it's broken!"

What it really needs is someone within Apple to come to that conclusion. Maybe it's a developer being asked to fix a bug who can stand up and say "no more - we do this properly" and then we need someone high up to listen and allow the time to get it done.

If Apple said that they were rewriting VoiceOver on the Mac. It would mean no new features for a couple of years, but it would provide a stable platform going forwards, then I think many of us would be pretty excited about that.

Right now it feels like we are all a bit stuck.

I'm not sure if extracting VoiceOver from MacOs would help or not. As has been said, it's probably not so much VoiceOver as the underlying stuff that it uses.

Thanks for the clarification. The thing that got me thinking about the provided accessibility information not to be unique in AppKit, for example, is that you've said that some system services do not respond properly to AX requests.

What I meant with that was that some first-party system services that have open connections to the window server but run mostly in the background with accessory or restricted activation policies accept accessibility messages but don't actually reply to them, and there are quite a few of them. This means that, if you configure the accessibility API to wait 5 seconds for a reply before timing out, which is a pretty normal configuration for a screen-reader, every time you talk to one of those services you are forced to wait 5 seconds before you can use the accessibility API again, because it's not thread-safe and is also blocking, so until it unblocks by timing out you cannot do anything else with it. In this case this is possible because we're talking about Apple's own services that are likely using private APIs that might not be correctly implemented, though this is pure speculation since I've never attempted to chase down this problem.

In order for a screen-reader to work properly and have the ability to focus system alerts as well as special windows like the Command+Tab and Dock views, the permission request and notification alerts, the Mission Control, App Launcher, Exposé, Time Machine, Force Quit Application window, and many other special views from applications or services that cannot be activated through the Command+Tab menu or the Dock, it must establish an accessibility connection to everything that also connects to the window server, not just regular applications. The problem is that, by not replying to the accessibility messages, the misbehaving system services coupled with the blocking API end up stalling the screen-reader's initialization process, and since they are quite a few of them, these delays add up and create a usability problem for users.

The workaround that I was planning on using, and is also likely to be what Apple does in VoiceOver, is to black list all the bundle identifiers belonging to services or applications that are known to not behave properly, which might provide a temporary relief but doesn't address the root problem. VoiceOver seems to be full of workarounds like this, and is also quite unstable, so beyond all the problems affecting the accessibility infrastructure and Cocoa, VoiceOver also contributes to the party with some problems of its own, so everything is broken.

The best analogy I can make regarding my thoughts on this matter is that the Titanic is sinking, and the captain is ordering the ship's crew to frantically scoop up as much water as they can with some buckets and throw it overboard. I mean yeah it might delay the sinking a little bit, but water keeps coming in, so the problem isn't really getting solved.

Again, thank you for the deep insights that you gave on this topic. I greatly appreciate that! I wonder whether the state of Accessibility is better on iOS and all the forks like watchOS and tvOS? At least from a user's perspective, those systems behave much more reliable than macOS does regarding VoiceOver.

I can't imagine how they are even able to try keep up with the modern web and screen reader accessibility, it must be a real nightmare.

I suspect most screen readers have plenty of work-arounds attempting to deal with bugs in system components and applications.

The Orca screen reader for Linux recently underwent revision to make it more maintainable and to reduce the number of work-arounds for various bugs. A prototype was also created for a new accessibility architecture that holds considerable promise.

Apple has vastly more resources and should be able to run similar projects for macOS.

Hi,

I have not tryed orca. question: is it simple to learn? and can you use it in UTM on MacOS? I would love to try it some day in future.

On the first question: I've never had difficulty learning any screen reader, so I can't say how hard Orca is to learn. For smart people who understand technology, it should be easy enough.

On the second question, it should run in a virtual machine under macOS, but I haven't tried it. My desktop Linux systems have all run directly on the hardware - no virtualization involved.