Re: Introducing support for selective UIA event registration in NVDA Alpha snapshots
Hi,toggle quoted messageShow quoted text
Re-initializing UIA handler thread is a performance nightmare - causes NVDA to freeze until the thread comes to life.
There are two possible routes I've investigated, both involving event handlers and a pointer to UIA element for the object in question (obj.UIAElement):
1. Re-register local event handler group for the specific element by calling UIAHandler.handler.removeEventHandlerGroup then UIAHandler.handler.addEventHandlerGroup in that order for each UIA event the object needs to deal with. Code that needs to run on NVDA 2020.2 beta and earlier must check existence of "addEventHandlerGroup" in UIA handler thread and see if selective event registration is enabled. But that's not quite flexible.
2. Instruct UIA handler thread to listen to events for the specific element: this can be done by calling UIAHandler.handler.addLocalEventHandlerGroupToElement. The good news is that unlike option 1, you don't have to specify exactly which event handler group you want to register the element with. The big downside is that if an application generates new UIA elements (say, when a list opens each time something happens), calling UIAHandler.handler.removeLocalEventHandlerGroupToElement will NOT remove elements in the internal elements set - you must clean them up by cycling through the set of locally registered elements by criteria such as process ID, automation ID, UIA class name and such, followed by removing found elements (process ID is more reliable than automation ID because multiple processes may come with elements proclaiming the same automation ID; this isn't recommended but sometimes this is seen with some Windows 10 apps, even those from Microsoft).
For folks who may not know what Leonard, Bill, and I are talking about:
An event handler group is a collection of events for a UIA element. When a UI Automation client such as NVDA starts, it informs UIAutomationCore.dll that the client needs to listen to events coming from any UIA element, ranging from the currently focused element to the whole UIA tree (everything). In the past, a UIA client would register event handlers one at a time. Starting with Windows 10 Version 1809 (October 2018 Update) and Server 2019, a more convenient way to organize events called "event handler group" was introduced (part of IUIAutomation6 interface). The biggest benefit is that rather than registering one event handler at a time, multiple event handlers can be registered at once, thereby improving performance (unless UIAutomationCore.dll itself is stalled).
As part of his work, Leonard added event handler group support in UIA handler (basically a COM thread). But knowing that many users are still using older Windows releases (Windows 7 Service Pack 1, Windows 8.x, Windows 10 prior to Version 1809), a fake event handler group class is introduced to simulate the actual IUIAutomation6::EventHandlerGroup behavior: trying to register multiple events at once. The code in question goes one step further by teaching NVDA to listen to events coming from system focus and its ancestors only (parent object, or the route from the desktop object to the foreground object, and in turn from the foreground window to the focused control). Although it does introduce performance benefits, it needs to be validated by further testing by testers.
The ideal testing scenarios would be:
* Traveling through UIA universes i.e. testing apps such as Visual Studio where many UIA events are fired by many things at once.
* Reading overlays i.e. modern input features, touch keyboard, among others.
* No regressions, especially for people using Windows 7 and 8.1 (not Windows 8.0 because it is end of life since January 2016); although not recommended, you should test the code on unsupported Windows 10 releases (1507, 1511, 1607, 1703) if needed.
The way I test this code is (note: you don't have to follow exactly what I'm doing):
1. Enable selective UIA event registration. YOU MUST SAVE SETTINGS AND RESTART NVDA AFTER DOING SO! Without this, UIA handler thread will not be able to enable event handler group support (this happens early in the handler thread initialization). An alternative way to do this without restarting NVDA completely is opening Python Console and typing "import UIAHandler; UIAHandler.terminate(); UIAHandler.initialize()" with each part on separate lines.
2. Use NVDA as normal.
3. If you do encounter issues, YOU MUST RECORD WHAT NVDA IS DOING! Please do record what you are doing and how NVDA responds versus what it should do (either mentally, electronically, physically).
4. Turn off selective UIA event registration (again, RESTART NVDA!).
5. Perform the tasks that would trigger an error in step 3. Compare and record what NVDA is doing now versus what it did with selective event registration enabled.
6. If the problem is indeed caused by selective UIA registration, do let us know immediately (here or on GitHub). Please provide NVDA version, Windows version, how you are reproducing the issue (complete with app name and version, keyboard commands if applicable), and how NVDA responded (speech, braille, etc.). Although folks might be able to guess correctly, more details would help us debug faster, make NVDA more reliable, open up more test cases, and sleep better at nights (programming and debugging all night isn't fun, folks).
Hope this clarifies a lot of things.
From: email@example.com <firstname.lastname@example.org> On Behalf Of Leonard de Ruijter
Sent: Monday, July 6, 2020 10:59 PM
Subject: Re: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots
The problem here is that it isn't very easy to switch between the global and local event registration. It requires removing all UIA event handlers and registering them again, which somehow fails, probably due to a comtypes bug. Alternatively, it could be possible to re-initialize the UIAHandler, but even of that I'm not sure, I recall that was also causing issues for me.
On 06/07/2020 22:20, Joseph Lee wrote: