Topics

Introducing support for selective UIA event registration in NVDA Alpha snapshots

 

Dear all,


As many of you may know, people using Microsoft Visual Studio or other applications mainly accessed by UIAutomation can experience some major lag while using them with NVDA. In an attempt to fix this, I've spent some hours in coming up with an approach that majorly improves performance in these applications. Thanks to great feedback from NV Access and them merging it, it can now be enabled in most recent NVDA Alpha snapshots by enabling the "Enable selective registration for UI Automation events and property changes" option in NVDA's advanced settings.


Taking Visual Studio as an example, with this option enabled, I notice major performance improvements in the following areas:

* Nuget package manager

* Loading/moving through the error list

* Options like go to base, go to implementation, go to definition. etc.

Performance improvements aren't limited to that though. To go a bit technical, rather than requesting for all UIA events and processing/discarding them in NVDA's process, we're now mostly only requesting events for objects of interest. This means that many events fired on other controls than the focused control aren't simply discarded by NVDA, but rather not even seen by NVDA at all, thereby taking out the processing time needed to discard them.

As there might be some regressions introduced by this feature, it is marked advanced/experimental. As I hope it eventually will become the default, I'd really appreciate it if you could report any bugs or regressions perceived with this feature enabled on NVDA's Github.


Kind regards,

Leonard

 

Hello everyone,
Folks on NVDA add-ons list may have seen the below message, but just to reiterate:
For people using Windows 10 App Essentials: you'll notice that if you do enable the option described here, you won't be able to hear emoji panel changes as you scroll through emoji categories. I'm testing a potential temporary fix, which will be rolling out to development snapshot users soon.
As noted by Leonard, please do file issues on GitHub should you notice regressions.
Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard de Ruijter
Sent: Sunday, July 5, 2020 11:11 PM
To: nvda-devel@groups.io; program-l@...
Subject: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Dear all,


As many of you may know, people using Microsoft Visual Studio or other applications mainly accessed by UIAutomation can experience some major lag while using them with NVDA. In an attempt to fix this, I've spent some hours in coming up with an approach that majorly improves performance in these applications. Thanks to great feedback from NV Access and them merging it, it can now be enabled in most recent NVDA Alpha snapshots by enabling the "Enable selective registration for UI Automation events and property changes" option in NVDA's advanced settings.


Taking Visual Studio as an example, with this option enabled, I notice major performance improvements in the following areas:

* Nuget package manager

* Loading/moving through the error list

* Options like go to base, go to implementation, go to definition. etc.

Performance improvements aren't limited to that though. To go a bit technical, rather than requesting for all UIA events and processing/discarding them in NVDA's process, we're now mostly only requesting events for objects of interest. This means that many events fired on other controls than the focused control aren't simply discarded by NVDA, but rather not even seen by NVDA at all, thereby taking out the processing time needed to discard them.

As there might be some regressions introduced by this feature, it is marked advanced/experimental. As I hope it eventually will become the default, I'd really appreciate it if you could report any bugs or regressions perceived with this feature enabled on NVDA's Github.


Kind regards,

Leonard

Bill Dengler
 

Hi,
Is it possible to disable this on a per-process level?
I've written an app module (a self-contained version of #10910) that needs to react to background windows from the app.

Thanks,
Bill

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Joseph Lee
Sent: Monday, 6 July 2020 02:20
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Hello everyone,
Folks on NVDA add-ons list may have seen the below message, but just to reiterate:
For people using Windows 10 App Essentials: you'll notice that if you do enable the option described here, you won't be able to hear emoji panel changes as you scroll through emoji categories. I'm testing a potential temporary fix, which will be rolling out to development snapshot users soon.
As noted by Leonard, please do file issues on GitHub should you notice regressions.
Cheers,
Joseph


-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard de Ruijter
Sent: Sunday, July 5, 2020 11:11 PM
To: nvda-devel@groups.io; program-l@...
Subject: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Dear all,


As many of you may know, people using Microsoft Visual Studio or other applications mainly accessed by UIAutomation can experience some major lag while using them with NVDA. In an attempt to fix this, I've spent some hours in coming up with an approach that majorly improves performance in these applications. Thanks to great feedback from NV Access and them merging it, it can now be enabled in most recent NVDA Alpha snapshots by enabling the "Enable selective registration for UI Automation events and property changes" option in NVDA's advanced settings.


Taking Visual Studio as an example, with this option enabled, I notice major performance improvements in the following areas:

* Nuget package manager

* Loading/moving through the error list

* Options like go to base, go to implementation, go to definition. etc.

Performance improvements aren't limited to that though. To go a bit technical, rather than requesting for all UIA events and processing/discarding them in NVDA's process, we're now mostly only requesting events for objects of interest. This means that many events fired on other controls than the focused control aren't simply discarded by NVDA, but rather not even seen by NVDA at all, thereby taking out the processing time needed to discard them.

As there might be some regressions introduced by this feature, it is marked advanced/experimental. As I hope it eventually will become the default, I'd really appreciate it if you could report any bugs or regressions perceived with this feature enabled on NVDA's Github.


Kind regards,

Leonard

 

Hi,
Not easy, but it can be done at the event handler level (that's the route I'm taking with modern input features at the moment).
Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Bill Dengler
Sent: Monday, July 6, 2020 1:09 PM
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Hi,
Is it possible to disable this on a per-process level?
I've written an app module (a self-contained version of #10910) that needs to react to background windows from the app.

Thanks,
Bill
-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Joseph Lee
Sent: Monday, 6 July 2020 02:20
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Hello everyone,
Folks on NVDA add-ons list may have seen the below message, but just to reiterate:
For people using Windows 10 App Essentials: you'll notice that if you do enable the option described here, you won't be able to hear emoji panel changes as you scroll through emoji categories. I'm testing a potential temporary fix, which will be rolling out to development snapshot users soon.
As noted by Leonard, please do file issues on GitHub should you notice regressions.
Cheers,
Joseph


-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard de Ruijter
Sent: Sunday, July 5, 2020 11:11 PM
To: nvda-devel@groups.io; program-l@...
Subject: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Dear all,


As many of you may know, people using Microsoft Visual Studio or other applications mainly accessed by UIAutomation can experience some major lag while using them with NVDA. In an attempt to fix this, I've spent some hours in coming up with an approach that majorly improves performance in these applications. Thanks to great feedback from NV Access and them merging it, it can now be enabled in most recent NVDA Alpha snapshots by enabling the "Enable selective registration for UI Automation events and property changes" option in NVDA's advanced settings.


Taking Visual Studio as an example, with this option enabled, I notice major performance improvements in the following areas:

* Nuget package manager

* Loading/moving through the error list

* Options like go to base, go to implementation, go to definition. etc.

Performance improvements aren't limited to that though. To go a bit technical, rather than requesting for all UIA events and processing/discarding them in NVDA's process, we're now mostly only requesting events for objects of interest. This means that many events fired on other controls than the focused control aren't simply discarded by NVDA, but rather not even seen by NVDA at all, thereby taking out the processing time needed to discard them.

As there might be some regressions introduced by this feature, it is marked advanced/experimental. As I hope it eventually will become the default, I'd really appreciate it if you could report any bugs or regressions perceived with this feature enabled on NVDA's Github.


Kind regards,

Leonard

 

Hey,


The problem here is that it isn't very easy to switch between the global and local event registration. It requires removing all UIA event handlers and registering them again, which somehow fails, probably due to a comtypes bug. Alternatively, it could be possible to re-initialize the UIAHandler, but even of that I'm not sure, I recall that was also causing issues for me.


Regards,

Leonard

On 06/07/2020 22:20, Joseph Lee wrote:
Hi,
Not easy, but it can be done at the event handler level (that's the route I'm taking with modern input features at the moment).
Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Bill Dengler
Sent: Monday, July 6, 2020 1:09 PM
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Hi,
Is it possible to disable this on a per-process level?
I've written an app module (a self-contained version of #10910) that needs to react to background windows from the app.

Thanks,
Bill
-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Joseph Lee
Sent: Monday, 6 July 2020 02:20
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Hello everyone,
Folks on NVDA add-ons list may have seen the below message, but just to reiterate:
For people using Windows 10 App Essentials: you'll notice that if you do enable the option described here, you won't be able to hear emoji panel changes as you scroll through emoji categories. I'm testing a potential temporary fix, which will be rolling out to development snapshot users soon.
As noted by Leonard, please do file issues on GitHub should you notice regressions.
Cheers,
Joseph


-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard de Ruijter
Sent: Sunday, July 5, 2020 11:11 PM
To: nvda-devel@groups.io; program-l@...
Subject: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Dear all,


As many of you may know, people using Microsoft Visual Studio or other applications mainly accessed by UIAutomation can experience some major lag while using them with NVDA. In an attempt to fix this, I've spent some hours in coming up with an approach that majorly improves performance in these applications. Thanks to great feedback from NV Access and them merging it, it can now be enabled in most recent NVDA Alpha snapshots by enabling the "Enable selective registration for UI Automation events and property changes" option in NVDA's advanced settings.


Taking Visual Studio as an example, with this option enabled, I notice major performance improvements in the following areas:

* Nuget package manager

* Loading/moving through the error list

* Options like go to base, go to implementation, go to definition. etc.

Performance improvements aren't limited to that though. To go a bit technical, rather than requesting for all UIA events and processing/discarding them in NVDA's process, we're now mostly only requesting events for objects of interest. This means that many events fired on other controls than the focused control aren't simply discarded by NVDA, but rather not even seen by NVDA at all, thereby taking out the processing time needed to discard them.

As there might be some regressions introduced by this feature, it is marked advanced/experimental. As I hope it eventually will become the default, I'd really appreciate it if you could report any bugs or regressions perceived with this feature enabled on NVDA's Github.


Kind regards,

Leonard












 

Hi,
Re-initializing UIA handler thread is a performance nightmare - causes NVDA to freeze until the thread comes to life.
There are two possible routes I've investigated, both involving event handlers and a pointer to UIA element for the object in question (obj.UIAElement):
1. Re-register local event handler group for the specific element by calling UIAHandler.handler.removeEventHandlerGroup then UIAHandler.handler.addEventHandlerGroup in that order for each UIA event the object needs to deal with. Code that needs to run on NVDA 2020.2 beta and earlier must check existence of "addEventHandlerGroup" in UIA handler thread and see if selective event registration is enabled. But that's not quite flexible.
2. Instruct UIA handler thread to listen to events for the specific element: this can be done by calling UIAHandler.handler.addLocalEventHandlerGroupToElement. The good news is that unlike option 1, you don't have to specify exactly which event handler group you want to register the element with. The big downside is that if an application generates new UIA elements (say, when a list opens each time something happens), calling UIAHandler.handler.removeLocalEventHandlerGroupToElement will NOT remove elements in the internal elements set - you must clean them up by cycling through the set of locally registered elements by criteria such as process ID, automation ID, UIA class name and such, followed by removing found elements (process ID is more reliable than automation ID because multiple processes may come with elements proclaiming the same automation ID; this isn't recommended but sometimes this is seen with some Windows 10 apps, even those from Microsoft).
For folks who may not know what Leonard, Bill, and I are talking about:
An event handler group is a collection of events for a UIA element. When a UI Automation client such as NVDA starts, it informs UIAutomationCore.dll that the client needs to listen to events coming from any UIA element, ranging from the currently focused element to the whole UIA tree (everything). In the past, a UIA client would register event handlers one at a time. Starting with Windows 10 Version 1809 (October 2018 Update) and Server 2019, a more convenient way to organize events called "event handler group" was introduced (part of IUIAutomation6 interface). The biggest benefit is that rather than registering one event handler at a time, multiple event handlers can be registered at once, thereby improving performance (unless UIAutomationCore.dll itself is stalled).
As part of his work, Leonard added event handler group support in UIA handler (basically a COM thread). But knowing that many users are still using older Windows releases (Windows 7 Service Pack 1, Windows 8.x, Windows 10 prior to Version 1809), a fake event handler group class is introduced to simulate the actual IUIAutomation6::EventHandlerGroup behavior: trying to register multiple events at once. The code in question goes one step further by teaching NVDA to listen to events coming from system focus and its ancestors only (parent object, or the route from the desktop object to the foreground object, and in turn from the foreground window to the focused control). Although it does introduce performance benefits, it needs to be validated by further testing by testers.
The ideal testing scenarios would be:
* Traveling through UIA universes i.e. testing apps such as Visual Studio where many UIA events are fired by many things at once.
* Reading overlays i.e. modern input features, touch keyboard, among others.
* No regressions, especially for people using Windows 7 and 8.1 (not Windows 8.0 because it is end of life since January 2016); although not recommended, you should test the code on unsupported Windows 10 releases (1507, 1511, 1607, 1703) if needed.

The way I test this code is (note: you don't have to follow exactly what I'm doing):
1. Enable selective UIA event registration. YOU MUST SAVE SETTINGS AND RESTART NVDA AFTER DOING SO! Without this, UIA handler thread will not be able to enable event handler group support (this happens early in the handler thread initialization). An alternative way to do this without restarting NVDA completely is opening Python Console and typing "import UIAHandler; UIAHandler.terminate(); UIAHandler.initialize()" with each part on separate lines.
2. Use NVDA as normal.
3. If you do encounter issues, YOU MUST RECORD WHAT NVDA IS DOING! Please do record what you are doing and how NVDA responds versus what it should do (either mentally, electronically, physically).
4. Turn off selective UIA event registration (again, RESTART NVDA!).
5. Perform the tasks that would trigger an error in step 3. Compare and record what NVDA is doing now versus what it did with selective event registration enabled.
6. If the problem is indeed caused by selective UIA registration, do let us know immediately (here or on GitHub). Please provide NVDA version, Windows version, how you are reproducing the issue (complete with app name and version, keyboard commands if applicable), and how NVDA responded (speech, braille, etc.). Although folks might be able to guess correctly, more details would help us debug faster, make NVDA more reliable, open up more test cases, and sleep better at nights (programming and debugging all night isn't fun, folks).
Hope this clarifies a lot of things.
Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard de Ruijter
Sent: Monday, July 6, 2020 10:59 PM
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Hey,


The problem here is that it isn't very easy to switch between the global and local event registration. It requires removing all UIA event handlers and registering them again, which somehow fails, probably due to a comtypes bug. Alternatively, it could be possible to re-initialize the UIAHandler, but even of that I'm not sure, I recall that was also causing issues for me.


Regards,

Leonard

On 06/07/2020 22:20, Joseph Lee wrote:
Hi,
Not easy, but it can be done at the event handler level (that's the route I'm taking with modern input features at the moment).
Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Bill
Dengler
Sent: Monday, July 6, 2020 1:09 PM
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event
registration in NVDA Alpha snapshots

Hi,
Is it possible to disable this on a per-process level?
I've written an app module (a self-contained version of #10910) that needs to react to background windows from the app.

Thanks,
Bill
-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Joseph
Lee
Sent: Monday, 6 July 2020 02:20
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event
registration in NVDA Alpha snapshots

Hello everyone,
Folks on NVDA add-ons list may have seen the below message, but just to reiterate:
For people using Windows 10 App Essentials: you'll notice that if you do enable the option described here, you won't be able to hear emoji panel changes as you scroll through emoji categories. I'm testing a potential temporary fix, which will be rolling out to development snapshot users soon.
As noted by Leonard, please do file issues on GitHub should you notice regressions.
Cheers,
Joseph


-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard
de Ruijter
Sent: Sunday, July 5, 2020 11:11 PM
To: nvda-devel@groups.io; program-l@...
Subject: [nvda-devel] Introducing support for selective UIA event
registration in NVDA Alpha snapshots

Dear all,


As many of you may know, people using Microsoft Visual Studio or other applications mainly accessed by UIAutomation can experience some major lag while using them with NVDA. In an attempt to fix this, I've spent some hours in coming up with an approach that majorly improves performance in these applications. Thanks to great feedback from NV Access and them merging it, it can now be enabled in most recent NVDA Alpha snapshots by enabling the "Enable selective registration for UI Automation events and property changes" option in NVDA's advanced settings.


Taking Visual Studio as an example, with this option enabled, I notice major performance improvements in the following areas:

* Nuget package manager

* Loading/moving through the error list

* Options like go to base, go to implementation, go to definition. etc.

Performance improvements aren't limited to that though. To go a bit technical, rather than requesting for all UIA events and processing/discarding them in NVDA's process, we're now mostly only requesting events for objects of interest. This means that many events fired on other controls than the focused control aren't simply discarded by NVDA, but rather not even seen by NVDA at all, thereby taking out the processing time needed to discard them.

As there might be some regressions introduced by this feature, it is marked advanced/experimental. As I hope it eventually will become the default, I'd really appreciate it if you could report any bugs or regressions perceived with this feature enabled on NVDA's Github.


Kind regards,

Leonard













Marlon Brandão de Sousa
 

Looking at what you are doing it seems that knowing deterministicaly all possible impacts in all apps will be very hard.


Is there a way to port the selective UIA event registration option to a per profile configuration?


If it is possible then the feature can be safely distributed because it can be turned off for applications that need the full registration and turned on for others, with all the associated performance problems you described here.

On 07/07/2020 03:53, Joseph Lee wrote:
Hi,
Re-initializing UIA handler thread is a performance nightmare - causes NVDA to freeze until the thread comes to life.
There are two possible routes I've investigated, both involving event handlers and a pointer to UIA element for the object in question (obj.UIAElement):
1. Re-register local event handler group for the specific element by calling UIAHandler.handler.removeEventHandlerGroup then UIAHandler.handler.addEventHandlerGroup in that order for each UIA event the object needs to deal with. Code that needs to run on NVDA 2020.2 beta and earlier must check existence of "addEventHandlerGroup" in UIA handler thread and see if selective event registration is enabled. But that's not quite flexible.
2. Instruct UIA handler thread to listen to events for the specific element: this can be done by calling UIAHandler.handler.addLocalEventHandlerGroupToElement. The good news is that unlike option 1, you don't have to specify exactly which event handler group you want to register the element with. The big downside is that if an application generates new UIA elements (say, when a list opens each time something happens), calling UIAHandler.handler.removeLocalEventHandlerGroupToElement will NOT remove elements in the internal elements set - you must clean them up by cycling through the set of locally registered elements by criteria such as process ID, automation ID, UIA class name and such, followed by removing found elements (process ID is more reliable than automation ID because multiple processes may come with elements proclaiming the same automation ID; this isn't recommended but sometimes this is seen with some Windows 10 apps, even those from Microsoft).
For folks who may not know what Leonard, Bill, and I are talking about:
An event handler group is a collection of events for a UIA element. When a UI Automation client such as NVDA starts, it informs UIAutomationCore.dll that the client needs to listen to events coming from any UIA element, ranging from the currently focused element to the whole UIA tree (everything). In the past, a UIA client would register event handlers one at a time. Starting with Windows 10 Version 1809 (October 2018 Update) and Server 2019, a more convenient way to organize events called "event handler group" was introduced (part of IUIAutomation6 interface). The biggest benefit is that rather than registering one event handler at a time, multiple event handlers can be registered at once, thereby improving performance (unless UIAutomationCore.dll itself is stalled).
As part of his work, Leonard added event handler group support in UIA handler (basically a COM thread). But knowing that many users are still using older Windows releases (Windows 7 Service Pack 1, Windows 8.x, Windows 10 prior to Version 1809), a fake event handler group class is introduced to simulate the actual IUIAutomation6::EventHandlerGroup behavior: trying to register multiple events at once. The code in question goes one step further by teaching NVDA to listen to events coming from system focus and its ancestors only (parent object, or the route from the desktop object to the foreground object, and in turn from the foreground window to the focused control). Although it does introduce performance benefits, it needs to be validated by further testing by testers.
The ideal testing scenarios would be:
* Traveling through UIA universes i.e. testing apps such as Visual Studio where many UIA events are fired by many things at once.
* Reading overlays i.e. modern input features, touch keyboard, among others.
* No regressions, especially for people using Windows 7 and 8.1 (not Windows 8.0 because it is end of life since January 2016); although not recommended, you should test the code on unsupported Windows 10 releases (1507, 1511, 1607, 1703) if needed.

The way I test this code is (note: you don't have to follow exactly what I'm doing):
1. Enable selective UIA event registration. YOU MUST SAVE SETTINGS AND RESTART NVDA AFTER DOING SO! Without this, UIA handler thread will not be able to enable event handler group support (this happens early in the handler thread initialization). An alternative way to do this without restarting NVDA completely is opening Python Console and typing "import UIAHandler; UIAHandler.terminate(); UIAHandler.initialize()" with each part on separate lines.
2. Use NVDA as normal.
3. If you do encounter issues, YOU MUST RECORD WHAT NVDA IS DOING! Please do record what you are doing and how NVDA responds versus what it should do (either mentally, electronically, physically).
4. Turn off selective UIA event registration (again, RESTART NVDA!).
5. Perform the tasks that would trigger an error in step 3. Compare and record what NVDA is doing now versus what it did with selective event registration enabled.
6. If the problem is indeed caused by selective UIA registration, do let us know immediately (here or on GitHub). Please provide NVDA version, Windows version, how you are reproducing the issue (complete with app name and version, keyboard commands if applicable), and how NVDA responded (speech, braille, etc.). Although folks might be able to guess correctly, more details would help us debug faster, make NVDA more reliable, open up more test cases, and sleep better at nights (programming and debugging all night isn't fun, folks).
Hope this clarifies a lot of things.
Cheers,
Joseph


-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard de Ruijter
Sent: Monday, July 6, 2020 10:59 PM
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Hey,


The problem here is that it isn't very easy to switch between the global and local event registration. It requires removing all UIA event handlers and registering them again, which somehow fails, probably due to a comtypes bug. Alternatively, it could be possible to re-initialize the UIAHandler, but even of that I'm not sure, I recall that was also causing issues for me.


Regards,

Leonard

On 06/07/2020 22:20, Joseph Lee wrote:
Hi,
Not easy, but it can be done at the event handler level (that's the route I'm taking with modern input features at the moment).
Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Bill
Dengler
Sent: Monday, July 6, 2020 1:09 PM
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event
registration in NVDA Alpha snapshots

Hi,
Is it possible to disable this on a per-process level?
I've written an app module (a self-contained version of #10910) that needs to react to background windows from the app.

Thanks,
Bill
-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Joseph
Lee
Sent: Monday, 6 July 2020 02:20
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event
registration in NVDA Alpha snapshots

Hello everyone,
Folks on NVDA add-ons list may have seen the below message, but just to reiterate:
For people using Windows 10 App Essentials: you'll notice that if you do enable the option described here, you won't be able to hear emoji panel changes as you scroll through emoji categories. I'm testing a potential temporary fix, which will be rolling out to development snapshot users soon.
As noted by Leonard, please do file issues on GitHub should you notice regressions.
Cheers,
Joseph


-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard
de Ruijter
Sent: Sunday, July 5, 2020 11:11 PM
To: nvda-devel@groups.io; program-l@...
Subject: [nvda-devel] Introducing support for selective UIA event
registration in NVDA Alpha snapshots

Dear all,


As many of you may know, people using Microsoft Visual Studio or other applications mainly accessed by UIAutomation can experience some major lag while using them with NVDA. In an attempt to fix this, I've spent some hours in coming up with an approach that majorly improves performance in these applications. Thanks to great feedback from NV Access and them merging it, it can now be enabled in most recent NVDA Alpha snapshots by enabling the "Enable selective registration for UI Automation events and property changes" option in NVDA's advanced settings.


Taking Visual Studio as an example, with this option enabled, I notice major performance improvements in the following areas:

* Nuget package manager

* Loading/moving through the error list

* Options like go to base, go to implementation, go to definition. etc.

Performance improvements aren't limited to that though. To go a bit technical, rather than requesting for all UIA events and processing/discarding them in NVDA's process, we're now mostly only requesting events for objects of interest. This means that many events fired on other controls than the focused control aren't simply discarded by NVDA, but rather not even seen by NVDA at all, thereby taking out the processing time needed to discard them.

As there might be some regressions introduced by this feature, it is marked advanced/experimental. As I hope it eventually will become the default, I'd really appreciate it if you could report any bugs or regressions perceived with this feature enabled on NVDA's Github.


Kind regards,

Leonard















 

Hey


Looking at what you are doing it seems that knowing deterministicaly all possible impacts in all apps will be very hard.
While this is true, I think the new approach more closely resembled what other screen readers like JAWS and Narrator do.
Is there a way to port the selective UIA event registration option to a per profile configuration?

I'm afraid not. This was my initial intent, but it wasn't possible as noted in this earlier thread.


On 07/07/2020 03:53, Joseph Lee wrote:
Hi,
Re-initializing UIA handler thread is a performance nightmare - causes NVDA to freeze until the thread comes to life.
There are two possible routes I've investigated, both involving event handlers and a pointer to UIA element for the object in question (obj.UIAElement):
1. Re-register local event handler group for the specific element by calling UIAHandler.handler.removeEventHandlerGroup then UIAHandler.handler.addEventHandlerGroup in that order for each UIA event the object needs to deal with. Code that needs to run on NVDA 2020.2 beta and earlier must check existence of "addEventHandlerGroup" in UIA handler thread and see if selective event registration is enabled. But that's not quite flexible.
2. Instruct UIA handler thread to listen to events for the specific element: this can be done by calling UIAHandler.handler.addLocalEventHandlerGroupToElement. The good news is that unlike option 1, you don't have to specify exactly which event handler group you want to register the element with. The big downside is that if an application generates new UIA elements (say, when a list opens each time something happens), calling UIAHandler.handler.removeLocalEventHandlerGroupToElement will NOT remove elements in the internal elements set - you must clean them up by cycling through the set of locally registered elements by criteria such as process ID, automation ID, UIA class name and such, followed by removing found elements (process ID is more reliable than automation ID because multiple processes may come with elements proclaiming the same automation ID; this isn't recommended but sometimes this is seen with some Windows 10 apps, even those from Microsoft).
For folks who may not know what Leonard, Bill, and I are talking about:
An event handler group is a collection of events for a UIA element. When a UI Automation client such as NVDA starts, it informs UIAutomationCore.dll that the client needs to listen to events coming from any UIA element, ranging from the currently focused element to the whole UIA tree (everything). In the past, a UIA client would register event handlers one at a time. Starting with Windows 10 Version 1809 (October 2018 Update) and Server 2019, a more convenient way to organize events called "event handler group" was introduced (part of IUIAutomation6 interface). The biggest benefit is that rather than registering one event handler at a time, multiple event handlers can be registered at once, thereby improving performance (unless UIAutomationCore.dll itself is stalled).
As part of his work, Leonard added event handler group support in UIA handler (basically a COM thread). But knowing that many users are still using older Windows releases (Windows 7 Service Pack 1, Windows 8.x, Windows 10 prior to Version 1809), a fake event handler group class is introduced to simulate the actual IUIAutomation6::EventHandlerGroup behavior: trying to register multiple events at once. The code in question goes one step further by teaching NVDA to listen to events coming from system focus and its ancestors only (parent object, or the route from the desktop object to the foreground object, and in turn from the foreground window to the focused control). Although it does introduce performance benefits, it needs to be validated by further testing by testers.
The ideal testing scenarios would be:
* Traveling through UIA universes i.e. testing apps such as Visual Studio where many UIA events are fired by many things at once.
* Reading overlays i.e. modern input features, touch keyboard, among others.
* No regressions, especially for people using Windows 7 and 8.1 (not Windows 8.0 because it is end of life since January 2016); although not recommended, you should test the code on unsupported Windows 10 releases (1507, 1511, 1607, 1703) if needed.

The way I test this code is (note: you don't have to follow exactly what I'm doing):
1. Enable selective UIA event registration. YOU MUST SAVE SETTINGS AND RESTART NVDA AFTER DOING SO! Without this, UIA handler thread will not be able to enable event handler group support (this happens early in the handler thread initialization). An alternative way to do this without restarting NVDA completely is opening Python Console and typing "import UIAHandler; UIAHandler.terminate(); UIAHandler.initialize()" with each part on separate lines.
  2. Use NVDA as normal.
3. If you do encounter issues, YOU MUST RECORD WHAT NVDA IS DOING! Please do record what you are doing and how NVDA responds versus what it should do (either mentally, electronically, physically).
4. Turn off selective UIA event registration (again, RESTART NVDA!).
5. Perform the tasks that would trigger an error in step 3. Compare and record what NVDA is doing now versus what it did with selective event registration enabled.
6. If the problem is indeed caused by selective UIA registration, do let us know immediately (here or on GitHub). Please provide NVDA version, Windows version, how you are reproducing the issue (complete with app name and version, keyboard commands if applicable), and how NVDA responded (speech, braille, etc.). Although folks might be able to guess correctly, more details would help us debug faster, make NVDA more reliable, open up more test cases, and sleep better at nights (programming and debugging all night isn't fun, folks).
Hope this clarifies a lot of things.
Cheers,
Joseph


-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard de Ruijter
Sent: Monday, July 6, 2020 10:59 PM
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event registration in NVDA Alpha snapshots

Hey,


The problem here is that it isn't very easy to switch between the global and local event registration. It requires removing all UIA event handlers and registering them again, which somehow fails, probably due to a comtypes bug. Alternatively, it could be possible to re-initialize the UIAHandler, but even of that I'm not sure, I recall that was also causing issues for me.


Regards,

Leonard

On 06/07/2020 22:20, Joseph Lee wrote:
Hi,
Not easy, but it can be done at the event handler level (that's the route I'm taking with modern input features at the moment).
Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Bill
Dengler
Sent: Monday, July 6, 2020 1:09 PM
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event
registration in NVDA Alpha snapshots

Hi,
Is it possible to disable this on a per-process level?
I've written an app module (a self-contained version of #10910) that needs to react to background windows from the app.

Thanks,
Bill
-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Joseph
Lee
Sent: Monday, 6 July 2020 02:20
To: nvda-devel@groups.io
Subject: Re: [nvda-devel] Introducing support for selective UIA event
registration in NVDA Alpha snapshots

Hello everyone,
Folks on NVDA add-ons list may have seen the below message, but just to reiterate:
For people using Windows 10 App Essentials: you'll notice that if you do enable the option described here, you won't be able to hear emoji panel changes as you scroll through emoji categories. I'm testing a potential temporary fix, which will be rolling out to development snapshot users soon.
As noted by Leonard, please do file issues on GitHub should you notice regressions.
Cheers,
Joseph


-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Leonard
de Ruijter
Sent: Sunday, July 5, 2020 11:11 PM
To: nvda-devel@groups.io; program-l@...
Subject: [nvda-devel] Introducing support for selective UIA event
registration in NVDA Alpha snapshots

Dear all,


As many of you may know, people using Microsoft Visual Studio or other applications mainly accessed by UIAutomation can experience some major lag while using them with NVDA. In an attempt to fix this, I've spent some hours in coming up with an approach that majorly improves performance in these applications. Thanks to great feedback from NV Access and them merging it, it can now be enabled in most recent NVDA Alpha snapshots by enabling the "Enable selective registration for UI Automation events and property changes" option in NVDA's advanced settings.


Taking Visual Studio as an example, with this option enabled, I notice major performance improvements in the following areas:

* Nuget package manager

* Loading/moving through the error list

* Options like go to base, go to implementation, go to definition. etc.

Performance improvements aren't limited to that though. To go a bit technical, rather than requesting for all UIA events and processing/discarding them in NVDA's process, we're now mostly only requesting events for objects of interest. This means that many events fired on other controls than the focused control aren't simply discarded by NVDA, but rather not even seen by NVDA at all, thereby taking out the processing time needed to discard them.

As there might be some regressions introduced by this feature, it is marked advanced/experimental. As I hope it eventually will become the default, I'd really appreciate it if you could report any bugs or regressions perceived with this feature enabled on NVDA's Github.


Kind regards,

Leonard