Topics

IAccessible or UIA objects?


Vincent Le Goff
 

Hi everyone,

This question isn't well-phrased, because I'm not sure how to ask it and I prefer to stay focused on my issue, though it might have more consequences.

In short, I'm beginning to play with NVDA's Python console.  I would like to create a basic appModule for the file explorer in which one could press NVDA+A to get the content of the address bar.  Nothing fancy, really, and I thought I would wrap it all up in less than an hour.  So I opened the file explorer, pressed CTRL + NVDA + Z and began exploring.

I located the address bar and the explorer window itself pretty quickly.  I thought I would just go from the foreground object, which can be obtained from the API, and then browse through its children.  On further inspection, I could determine that address_bar (my variable containing the address bar) did have the foreground object in its parents.  Something like that:

parent = address_bar
parents = []
while parent:
...    parents.append(parent)
...    parent = parent.parent
...
fg in parents
True

Good, I thought.  But what I was trying to do was the reverse (that is, have the foreground object as starting point and then find the address bar).  And that proved trickier:

address_bar in fg.recursiveDescendants
False

What now?  I actually browsed fg's children and children and children myself to make sure the result of recursiveDescendants was the one I expected.  address_bar wasn't in it.  In fact from the foreground object, the only descendants with role of 8 (edit) were various cells for the explorer table.  Nothing like an address bar.

I was pretty puzzled.  I remember on this list someone explaining why parent and children didn't always match as expected, but I can't remember the explanation.  On further debug, I noticed that my foreground object was an UIA, although my address bar was an IAccessible.  Now I thought IACcessible was to replace (or wrap) all objects, so that browsing by parents and children would only raise IAccessible objects.  But obviously I'm wrong and that's why I get this strange behavior.  My foreground's children didn't match the previous item in the address_bar parents, although one of them should be.  Their roles didn't match either.  So I'm pretty puzzled.

Why would we manipulate UIA instead of IAccessible?  They seem to have a close API, and I don't mean to say UIA isn't great, but the documentation seemed to point out IAccessible was to ensure a consistent tree among accessible libraries.  If someone can explain to me why parents and children do not match in this context, it would be great.

As a bonus, if anyone can tell me how I should locate a given object from a parent: I could browse all children and filter them (by role, maybe), but isn't there something better to do?  I would like my addon not to break too soon with Windows updates. Admittedly, explorer is somewhat stable these days, still, unique IDs or identifiers would be good!

Thanks in advance,

Vincent


 

Hi,
When it comes to object navigation (either done manually using keyboard commands or via code), NVDA doesn't care about the type of object as long as it can locate objects of a given relation. This is why you can move from an IAccessible object to a UIA child or look up an IAccessible object from a UIA coded foreground window. As far as object hierarchy representation is concerned, these objects are represented by window handles, and as far as actual representation is concerned, it's a double-word (dword/32-bit) integer.
To figure out if a window is a descendent of a foreground window, a more effective solution is calling windowUtils.findDescendentWindow with the following parameters filled in:
* Parent: the window handle of the foreground window.
* Control ID: the unique control ID for a window. This works better with IAccessible objects.
* Class name: the window class name for the object.

The above method works best for IAccessible objects with unique class names and control ID's inside a given app provided you know the window handle for the foreground window.

A slight variation exists in UIA (credit: Abdel from NVDA Add-ons list):
1. Locate the UIA client object.
2. Create a property condition that will look for a specific UIA property (Automation Id, for example) and the actual property string you are looking for.
3. Call clientObject:: ElementFromHandleBuildCache method, passing in the window handle for the foreground object and a base cache request. Be sure to save the result in a variable.
4. Look for the actual element by calling FindFirstBuildCache method on the variable you just saved, making sure to test if the variable is even usable. Be sure to tell UIA that you want to look up descendants, provide the condition, and ask for a base cache request. You must check what the actual element returned is to avoid passing in a NULL pointer.

For reference, the UIA method I described is implemented in Cortana app module, part of Windows 10 App Essentials add-on. Countless add-ons use findDescendentWindow method.

Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Vincent Le Goff
Sent: Monday, December 21, 2020 1:10 PM
To: nvda-devel@groups.io
Subject: [nvda-devel] IAccessible or UIA objects?

Hi everyone,

This question isn't well-phrased, because I'm not sure how to ask it and I prefer to stay focused on my issue, though it might have more consequences.

In short, I'm beginning to play with NVDA's Python console. I would like to create a basic appModule for the file explorer in which one could press NVDA+A to get the content of the address bar. Nothing fancy, really, and I thought I would wrap it all up in less than an hour. So I opened the file explorer, pressed CTRL + NVDA + Z and began exploring.

I located the address bar and the explorer window itself pretty quickly. I thought I would just go from the foreground object, which can be obtained from the API, and then browse through its children. On further inspection, I could determine that address_bar (my variable containing the address bar) did have the foreground object in its parents. Something like that:

>>> parent = address_bar
>>> parents = []
>>> while parent:
... parents.append(parent)
... parent = parent.parent
...
>>> fg in parents
True

Good, I thought. But what I was trying to do was the reverse (that is, have the foreground object as starting point and then find the address bar). And that proved trickier:

>>> address_bar in fg.recursiveDescendants False

What now? I actually browsed fg's children and children and children myself to make sure the result of recursiveDescendants was the one I expected. address_bar wasn't in it. In fact from the foreground object, the only descendants with role of 8 (edit) were various cells for the explorer table. Nothing like an address bar.

I was pretty puzzled. I remember on this list someone explaining why parent and children didn't always match as expected, but I can't remember the explanation. On further debug, I noticed that my foreground object was an UIA, although my address bar was an IAccessible. Now I thought IACcessible was to replace (or wrap) all objects, so that browsing by parents and children would only raise IAccessible objects. But obviously I'm wrong and that's why I get this strange behavior. My foreground's children didn't match the previous item in the address_bar parents, although one of them should be. Their roles didn't match either. So I'm pretty puzzled.

Why would we manipulate UIA instead of IAccessible? They seem to have a close API, and I don't mean to say UIA isn't great, but the documentation seemed to point out IAccessible was to ensure a consistent tree among accessible libraries. If someone can explain to me why parents and children do not match in this context, it would be great.

As a bonus, if anyone can tell me how I should locate a given object from a parent: I could browse all children and filter them (by role, maybe), but isn't there something better to do? I would like my addon not to break too soon with Windows updates. Admittedly, explorer is somewhat stable these days, still, unique IDs or identifiers would be good!

Thanks in advance,

Vincent


Vincent Le Goff
 

Hi Joseph,

Thanks for your answer.

I guess I'm still lacking in understanding.  I would be happy to avoid the low-level approach with ctypes altogether, though it doesn't seem I can't escape playing with windows as integers.  I could find the window handle in the foreground object (at least, I think so), since the aPI can fetch the foreground object, and then the windowHandle property does work.  As for the other arguments, I'm not even sure how to get them yet.  Control ID?  I heard about that once, and I thought all objects on the window had one, but I can't find it on the NVDA object's attributes.  But in any case, this method will return, presumably, another window handle (that is another int) which I'll need to somehow convert back to an IAccessible, or something similar.  I guess I'm stuck in backward thinking and playing with things I still don't fully understand.

I would guess I need to find an appModule that does what I want, specifically retrieving an object on the window and giving its state, no matter where the focus is (as long as the application itself is focused, of course).  Here I go hunting for one!

Thanks again,

Vincent

On 12/21/2020 10:45 PM, Joseph Lee wrote:
Hi,
When it comes to object navigation (either done manually using keyboard commands or via code), NVDA doesn't care about the type of object as long as it can locate objects of a given relation. This is why you can move from an IAccessible object to a UIA child or look up an IAccessible object from a UIA coded foreground window. As far as object hierarchy representation is concerned, these objects are represented by window handles, and as far as actual representation is concerned, it's a double-word (dword/32-bit) integer.
To figure out if a window is a descendent of a foreground window, a more effective solution is calling windowUtils.findDescendentWindow with the following parameters filled in:
* Parent: the window handle of the foreground window.
* Control ID: the unique control ID for a window. This works better with IAccessible objects.
* Class name: the window class name for the object.

The above method works best for IAccessible objects with unique class names and control ID's inside a given app provided you know the window handle for the foreground window.

A slight variation exists in UIA (credit: Abdel from NVDA Add-ons list):
1. Locate the UIA client object.
2. Create a property condition that will look for a specific UIA property (Automation Id, for example) and the actual property string you are looking for.
3. Call clientObject:: ElementFromHandleBuildCache method, passing in the window handle for the foreground object and a base cache request. Be sure to save the result in a variable.
4. Look for the actual element by calling FindFirstBuildCache method on the variable you just saved, making sure to test if the variable is even usable. Be sure to tell UIA that you want to look up descendants, provide the condition, and ask for a base cache request. You must check what the actual element returned is to avoid passing in a NULL pointer.

For reference, the UIA method I described is implemented in Cortana app module, part of Windows 10 App Essentials add-on. Countless add-ons use findDescendentWindow method.

Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Vincent Le Goff
Sent: Monday, December 21, 2020 1:10 PM
To: nvda-devel@groups.io
Subject: [nvda-devel] IAccessible or UIA objects?

Hi everyone,

This question isn't well-phrased, because I'm not sure how to ask it and I prefer to stay focused on my issue, though it might have more consequences.

In short, I'm beginning to play with NVDA's Python console. I would like to create a basic appModule for the file explorer in which one could press NVDA+A to get the content of the address bar. Nothing fancy, really, and I thought I would wrap it all up in less than an hour. So I opened the file explorer, pressed CTRL + NVDA + Z and began exploring.

I located the address bar and the explorer window itself pretty quickly. I thought I would just go from the foreground object, which can be obtained from the API, and then browse through its children. On further inspection, I could determine that address_bar (my variable containing the address bar) did have the foreground object in its parents. Something like that:

>>> parent = address_bar
>>> parents = []
>>> while parent:
... parents.append(parent)
... parent = parent.parent
...
>>> fg in parents
True

Good, I thought. But what I was trying to do was the reverse (that is, have the foreground object as starting point and then find the address bar). And that proved trickier:

>>> address_bar in fg.recursiveDescendants False

What now? I actually browsed fg's children and children and children myself to make sure the result of recursiveDescendants was the one I expected. address_bar wasn't in it. In fact from the foreground object, the only descendants with role of 8 (edit) were various cells for the explorer table. Nothing like an address bar.

I was pretty puzzled. I remember on this list someone explaining why parent and children didn't always match as expected, but I can't remember the explanation. On further debug, I noticed that my foreground object was an UIA, although my address bar was an IAccessible. Now I thought IACcessible was to replace (or wrap) all objects, so that browsing by parents and children would only raise IAccessible objects. But obviously I'm wrong and that's why I get this strange behavior. My foreground's children didn't match the previous item in the address_bar parents, although one of them should be. Their roles didn't match either. So I'm pretty puzzled.

Why would we manipulate UIA instead of IAccessible? They seem to have a close API, and I don't mean to say UIA isn't great, but the documentation seemed to point out IAccessible was to ensure a consistent tree among accessible libraries. If someone can explain to me why parents and children do not match in this context, it would be great.

As a bonus, if anyone can tell me how I should locate a given object from a parent: I could browse all children and filter them (by role, maybe), but isn't there something better to do? I would like my addon not to break too soon with Windows updates. Admittedly, explorer is somewhat stable these days, still, unique IDs or identifiers would be good!

Thanks in advance,

Vincent










Alberto Buffolino
 

Vincent Le Goff, il 22/12/2020 11.46, ha scritto:
I would guess I need to find an appModule that does what I want,
Alberto:
Hi Vincent,
if you're interested, ColumnsReview works extensively with explorer, mixing approaches from IAccessible and shell32 (that's often more reliable than UIA):
https://github.com/ABuffEr/columnsReview
Alberto


James Scholes
 

For what it's worth, I'm not convinced the object you need exists in the form you need it at all times. E.g. if I focus on a file list in Explorer, I can move up to the parent level, and then navigate seven times to the left to find the control called the "Address band toolbar". But this only contains the "previous locations" and "refresh buttons". Moving left again from that toolbar brings me to the "Address: <path>" control, which is also a toolbar, but this only contains buttons representing the individual path elements.

The second toolbar control does contain the full path in its accessible name, so you could grab it from there. But the actual editable address bar only seems to become present in the control order when I explicitly navigate to it, at which point it is positioned to the right of the "Address band toolbar" as a sibling. Note that all of this is with simple review enabled, and only based on an initial glance.

Regards,

James Scholes

On 22/12/2020 at 4:46 am, Vincent Le Goff wrote:
Hi Joseph,

Thanks for your answer.

I guess I'm still lacking in understanding.  I would be happy to avoid
the low-level approach with ctypes altogether, though it doesn't seem I
can't escape playing with windows as integers.  I could find the window
handle in the foreground object (at least, I think so), since the aPI
can fetch the foreground object, and then the windowHandle property does
work.  As for the other arguments, I'm not even sure how to get them
yet.  Control ID?  I heard about that once, and I thought all objects on
the window had one, but I can't find it on the NVDA object's
attributes.  But in any case, this method will return, presumably,
another window handle (that is another int) which I'll need to somehow
convert back to an IAccessible, or something similar.  I guess I'm stuck
in backward thinking and playing with things I still don't fully understand.

I would guess I need to find an appModule that does what I want,
specifically retrieving an object on the window and giving its state, no
matter where the focus is (as long as the application itself is focused,
of course).  Here I go hunting for one!

Thanks again,

Vincent

On 12/21/2020 10:45 PM, Joseph Lee wrote:
Hi,
When it comes to object navigation (either done manually using keyboard commands or via code), NVDA doesn't care about the type of object as long as it can locate objects of a given relation. This is why you can move from an IAccessible object to a UIA child or look up an IAccessible object from a UIA coded foreground window. As far as object hierarchy representation is concerned, these objects are represented by window handles, and as far as actual representation is concerned, it's a double-word (dword/32-bit) integer.
To figure out if a window is a descendent of a foreground window, a more effective solution is calling windowUtils.findDescendentWindow with the following parameters filled in:
* Parent: the window handle of the foreground window.
* Control ID: the unique control ID for a window. This works better with IAccessible objects.
* Class name: the window class name for the object.

The above method works best for IAccessible objects with unique class names and control ID's inside a given app provided you know the window handle for the foreground window.

A slight variation exists in UIA (credit: Abdel from NVDA Add-ons list):
1. Locate the UIA client object.
2. Create a property condition that will look for a specific UIA property (Automation Id, for example) and the actual property string you are looking for.
3. Call clientObject:: ElementFromHandleBuildCache method, passing in the window handle for the foreground object and a base cache request. Be sure to save the result in a variable.
4. Look for the actual element by calling FindFirstBuildCache method on the variable you just saved, making sure to test if the variable is even usable. Be sure to tell UIA that you want to look up descendants, provide the condition, and ask for a base cache request. You must check what the actual element returned is to avoid passing in a NULL pointer.

For reference, the UIA method I described is implemented in Cortana app module, part of Windows 10 App Essentials add-on. Countless add-ons use findDescendentWindow method.

Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Vincent Le Goff
Sent: Monday, December 21, 2020 1:10 PM
To: nvda-devel@groups.io
Subject: [nvda-devel] IAccessible or UIA objects?

Hi everyone,

This question isn't well-phrased, because I'm not sure how to ask it and I prefer to stay focused on my issue, though it might have more consequences.

In short, I'm beginning to play with NVDA's Python console. I would like to create a basic appModule for the file explorer in which one could press NVDA+A to get the content of the address bar. Nothing fancy, really, and I thought I would wrap it all up in less than an hour. So I opened the file explorer, pressed CTRL + NVDA + Z and began exploring.

I located the address bar and the explorer window itself pretty quickly. I thought I would just go from the foreground object, which can be obtained from the API, and then browse through its children. On further inspection, I could determine that address_bar (my variable containing the address bar) did have the foreground object in its parents. Something like that:

>>> parent = address_bar
>>> parents = []
>>> while parent:
... parents.append(parent)
... parent = parent.parent
...
>>> fg in parents
True

Good, I thought. But what I was trying to do was the reverse (that is, have the foreground object as starting point and then find the address bar). And that proved trickier:

>>> address_bar in fg.recursiveDescendants False

What now? I actually browsed fg's children and children and children myself to make sure the result of recursiveDescendants was the one I expected. address_bar wasn't in it. In fact from the foreground object, the only descendants with role of 8 (edit) were various cells for the explorer table. Nothing like an address bar.

I was pretty puzzled. I remember on this list someone explaining why parent and children didn't always match as expected, but I can't remember the explanation. On further debug, I noticed that my foreground object was an UIA, although my address bar was an IAccessible. Now I thought IACcessible was to replace (or wrap) all objects, so that browsing by parents and children would only raise IAccessible objects. But obviously I'm wrong and that's why I get this strange behavior. My foreground's children didn't match the previous item in the address_bar parents, although one of them should be. Their roles didn't match either. So I'm pretty puzzled.

Why would we manipulate UIA instead of IAccessible? They seem to have a close API, and I don't mean to say UIA isn't great, but the documentation seemed to point out IAccessible was to ensure a consistent tree among accessible libraries. If someone can explain to me why parents and children do not match in this context, it would be great.

As a bonus, if anyone can tell me how I should locate a given object from a parent: I could browse all children and filter them (by role, maybe), but isn't there something better to do? I would like my addon not to break too soon with Windows updates. Admittedly, explorer is somewhat stable these days, still, unique IDs or identifiers would be good!

Thanks in advance,

Vincent














Vincent Le Goff
 

Thanks to all.

So it seems I didn't have so many wrong ideas to begin with, but what you pointed out about the edit bar, I didn't get at first. Now I do, and I've found a workaround.

It seems there already is an appModule for explorer.exe, and defining my own appModule on top might not be a good idea, so I guess I should inherit from the default explorer appModule.  But everything seems to be easier, thanks again!

Cheers,

Vincent

On 12/22/2020 6:13 PM, James Scholes wrote:
For what it's worth, I'm not convinced the object you need exists in the form you need it at all times. E.g. if I focus on a file list in Explorer, I can move up to the parent level, and then navigate seven times to the left to find the control called the "Address band toolbar". But this only contains the "previous locations" and "refresh buttons". Moving left again from that toolbar brings me to the "Address: <path>" control, which is also a toolbar, but this only contains buttons representing the individual path elements.

The second toolbar control does contain the full path in its accessible name, so you could grab it from there. But the actual editable address bar only seems to become present in the control order when I explicitly navigate to it, at which point it is positioned to the right of the "Address band toolbar" as a sibling. Note that all of this is with simple review enabled, and only based on an initial glance.

Regards,

James Scholes

On 22/12/2020 at 4:46 am, Vincent Le Goff wrote:
Hi Joseph,

Thanks for your answer.

I guess I'm still lacking in understanding.  I would be happy to avoid
the low-level approach with ctypes altogether, though it doesn't seem I
can't escape playing with windows as integers.  I could find the window
handle in the foreground object (at least, I think so), since the aPI
can fetch the foreground object, and then the windowHandle property does
work.  As for the other arguments, I'm not even sure how to get them
yet.  Control ID?  I heard about that once, and I thought all objects on
the window had one, but I can't find it on the NVDA object's
attributes.  But in any case, this method will return, presumably,
another window handle (that is another int) which I'll need to somehow
convert back to an IAccessible, or something similar.  I guess I'm stuck
in backward thinking and playing with things I still don't fully understand.

I would guess I need to find an appModule that does what I want,
specifically retrieving an object on the window and giving its state, no
matter where the focus is (as long as the application itself is focused,
of course).  Here I go hunting for one!

Thanks again,

Vincent

On 12/21/2020 10:45 PM, Joseph Lee wrote:
Hi,
When it comes to object navigation (either done manually using keyboard commands or via code), NVDA doesn't care about the type of object as long as it can locate objects of a given relation. This is why you can move from an IAccessible object to a UIA child or look up an IAccessible object from a UIA coded foreground window. As far as object hierarchy representation is concerned, these objects are represented by window handles, and as far as actual representation is concerned, it's a double-word (dword/32-bit) integer.
To figure out if a window is a descendent of a foreground window, a more effective solution is calling windowUtils.findDescendentWindow with the following parameters filled in:
* Parent: the window handle of the foreground window.
* Control ID: the unique control ID for a window. This works better with IAccessible objects.
* Class name: the window class name for the object.

The above method works best for IAccessible objects with unique class names and control ID's inside a given app provided you know the window handle for the foreground window.

A slight variation exists in UIA (credit: Abdel from NVDA Add-ons list):
1. Locate the UIA client object.
2. Create a property condition that will look for a specific UIA property (Automation Id, for example) and the actual property string you are looking for.
3. Call clientObject:: ElementFromHandleBuildCache method, passing in the window handle for the foreground object and a base cache request. Be sure to save the result in a variable.
4. Look for the actual element by calling FindFirstBuildCache method on the variable you just saved, making sure to test if the variable is even usable. Be sure to tell UIA that you want to look up descendants, provide the condition, and ask for a base cache request. You must check what the actual element returned is to avoid passing in a NULL pointer.

For reference, the UIA method I described is implemented in Cortana app module, part of Windows 10 App Essentials add-on. Countless add-ons use findDescendentWindow method.

Cheers,
Joseph

-----Original Message-----
From: nvda-devel@groups.io <nvda-devel@groups.io> On Behalf Of Vincent Le Goff
Sent: Monday, December 21, 2020 1:10 PM
To: nvda-devel@groups.io
Subject: [nvda-devel] IAccessible or UIA objects?

Hi everyone,

This question isn't well-phrased, because I'm not sure how to ask it and I prefer to stay focused on my issue, though it might have more consequences.

In short, I'm beginning to play with NVDA's Python console. I would like to create a basic appModule for the file explorer in which one could press NVDA+A to get the content of the address bar. Nothing fancy, really, and I thought I would wrap it all up in less than an hour. So I opened the file explorer, pressed CTRL + NVDA + Z and began exploring.

I located the address bar and the explorer window itself pretty quickly. I thought I would just go from the foreground object, which can be obtained from the API, and then browse through its children. On further inspection, I could determine that address_bar (my variable containing the address bar) did have the foreground object in its parents. Something like that:

>>> parent = address_bar
>>> parents = []
>>> while parent:
... parents.append(parent)
... parent = parent.parent
...
>>> fg in parents
True

Good, I thought. But what I was trying to do was the reverse (that is, have the foreground object as starting point and then find the address bar). And that proved trickier:

>>> address_bar in fg.recursiveDescendants False

What now? I actually browsed fg's children and children and children myself to make sure the result of recursiveDescendants was the one I expected. address_bar wasn't in it. In fact from the foreground object, the only descendants with role of 8 (edit) were various cells for the explorer table. Nothing like an address bar.

I was pretty puzzled. I remember on this list someone explaining why parent and children didn't always match as expected, but I can't remember the explanation. On further debug, I noticed that my foreground object was an UIA, although my address bar was an IAccessible. Now I thought IACcessible was to replace (or wrap) all objects, so that browsing by parents and children would only raise IAccessible objects. But obviously I'm wrong and that's why I get this strange behavior. My foreground's children didn't match the previous item in the address_bar parents, although one of them should be. Their roles didn't match either. So I'm pretty puzzled.

Why would we manipulate UIA instead of IAccessible? They seem to have a close API, and I don't mean to say UIA isn't great, but the documentation seemed to point out IAccessible was to ensure a consistent tree among accessible libraries. If someone can explain to me why parents and children do not match in this context, it would be great.

As a bonus, if anyone can tell me how I should locate a given object from a parent: I could browse all children and filter them (by role, maybe), but isn't there something better to do? I would like my addon not to break too soon with Windows updates. Admittedly, explorer is somewhat stable these days, still, unique IDs or identifiers would be good!

Thanks in advance,

Vincent