Enabling automation for 3rd party controls by adding Accessibility

Posted by mgissing on Monday, October 26th, 2009 at 11:25 am to Improve Object Recognition

In many cases, custom Windows Forms or 3rd party provided controls are not built with accessibility in mind. It is often overlooked that by adding this functionality one does not only help those who are dependent on good accessibility, but that it also enables automation (and automated testing) of the control in question.

In the following example, I would like to show that in Windows Forms it is quite easy to add accessibility functions to an existing control, even to a rather complex control like a tree view. The control used in the example is the TreeViewAdv, a quite popular open source tree/list control. But please consider that also other complex controls like data grids or list views from other 3rd party vendors (Infragistics, DevExpress, Syncfusion, ComponentOne, Telerik, Janus, …) can be extended the same way as described within the following scenario.

Although it is quite nice to use, it lacks an accessibility implementation, rendering its content totally opaque to most automation tools. (Using Ranorex’ remote invocation capability, one can still extract data from the control, though.)

Here is an example of the control filled with some test data. You can see that in addition to a standard TreeView the TreeViewAdv can have columns as well:
TreeView Adv with example data

In its original form no information about the content of the control is available. To support automation, we are going to add the following accessibility features:

  • Every tree node is represented in the accessibility tree.
  • Every column cell is represented in the accessibility tree.
  • Every node or cell has geometry and visibility information.
  • Every node has a name (for identification) and a value (content for validation).

Let’s have a look at the necessary code. First, a reference to the “Accessibility” assembly, which is part of the .NET framework, has to be added. This assembly provides a managed wrapper and a number of utility classes around the COM-based Microsoft Active Accessibility (MSAA) layer.

MSAA has been around for some time now and is understood by most screen readers and automation tools. It has been superseded by UI Automation, which is especially suited for WPF applications, but still comes in handy for Windows Forms and MFC/ATL applications.

To provide a custom accessibility implementation for a control, the CreateAccessibilityInstance method has to be overridden and has to return a custom AccessibleObject implementation.

// overridden to return custom accessibility object instance
protected override AccessibleObject CreateAccessibilityInstance()
{
    return new TreeViewAdvAccessibleObject(this);
}

For our tree we will provide three classes, all deriving from AccessibleObject:

  1. TreeViewAdvAccessibleObject which represents the tree itself and acts as the top-level container. This actually derives from Control.ControlAccessibleObject
    which provides a basic accessibility implementation for control classes, e.g. for its location, size and visibility.
  2. AdvNodeAccessibleObject represents a tree node.
  3. AdvNodeCellAccessibleObject represents a cell of a certain node column.

Accessibility functionality is then implemented by overriding a number of properties and methods to provide useful information and functionality to an external application. Here is a short overview of the most important properties which need to be implemented:

  • Properties:
    • Role: An enumeration of different kinds of GUI objects, for example buttons, checkboxes, listitems, etc. In our example, we use Outline (which represents a tree) and OutlineItem (which represents a tree node) as well as Cell (which represents a table cell).
    • Name: A representative name for the object. It is usually used for (re-)identification purposes (e.g. “SaveAs” button), should not be content dependent, and might even be language-invariant.
    • Value: Represents the object’s content if it has any. For example the text in a text box, the Checked property of a check box, or the contents of a data cell.
    • State: A flag enumeration containing a number of states a GUI object can have, like Selected, Invisible, Expanded, Collapsed or Focused. In our example, Selected, Expanded and Collapsed are especially useful for tree nodes.
    • Bounds: A rectangle representing the bounding box of an object in screen coordinates. This property is very important for correct automation and capture/replay.
    • Parent: The parent of the accessible object. This must be implemented if the object is returned as a child of another accessible object (for consistency; see below).
  • Methods:
    • GetChild() and GetChildCount(): Returns a child/the number of children of the object. In our example, tree nodes return their child nodes and tables return cells as child objects. Make sure to implement the Parent property for objects returned by this functions!
    • Select(): Selects an object, if applicable. This method is a bit weird as it takes AccessibleSelection as a parameter which describes what kind of selection to perform. AccessibleSelection.Focus is very useful for automation if implemented, regardless whether an object is selectable or not.
    • HitTest(x,y): Returns the object at coordinates (x,y). This is useful for fast capture/replay and is easily implemented in our example, because TreeViewAdv already has built-in functionality for returning a tree node by coordinates.

Here is a part of the AdvNodeAccessibleObject implementation:

public class AdvNodeAccessibleObject : AccessibleObject
{
	TreeNodeAdv advNode;
	AdvNodeAccessibleObject parent;
	TreeViewAdvAccessibleObject owner;

	public AdvNodeAccessibleObject(TreeNodeAdv advNode, AdvNodeAccessibleObject parent,
									TreeViewAdvAccessibleObject owner) : base()
	{
		...
	}

	public override AccessibleRole Role
	{
		get { return AccessibleRole.OutlineItem; }
	}

	public override Rectangle Bounds
	{
		get
		{
			if (!advNode.IsVisible)
				return Rectangle.Empty;

                        // we need to convert client -> screen here
			Rectangle bounds = advNode.Tree.GetNodeBounds(advNode);
			Point p = advNode.Tree.ScrollPosition;

			int colHeaderY = advNode.Tree.UseColumns ? advNode.Tree.ColumnHeaderHeight : 0;

			bounds.Offset(-p.X, -p.Y * advNode.Tree.RowHeight + colHeaderY);
			return advNode.Tree.RectangleToScreen(bounds);
		}
	}

	public override string Name
	{
		get
		{
			foreach (NodeControlInfo info in advNode.Tree.GetNodeControls(advNode))
			{
				BindableControl ctrl = info.Control as BindableControl;
				if (ctrl != null)
				{
					string val = ctrl.GetValue(advNode) as string;
					if (val != null)
						return val;
				}
			}
			return null;
		}
	}

	public override AccessibleObject GetChild(int index)
	{
		int ctrlCount = advNode.Tree.Columns.Count;

		if (index < ctrlCount)
			return new AdvNodeCellAccessibleObject(advNode, index, this);
		else return new AdvNodeAccessibleObject(advNode.Nodes[index-ctrlCount], this, owner);
	}

	public override int GetChildCount()
	{
		return advNode.Nodes.Count + advNode.Tree.Columns.Count;
	}

	public override AccessibleObject Parent
	{
		get
		{
			if (advNode.Parent != advNode.Tree.Root)
				return parent != null ? parent : new AdvNodeAccessibleObject(advNode.Parent, null, owner);
			else return owner;
		}
	}

	...
}

This might look a bit complicated, but if you are familiar with the control you are extending, most accessibility properties and methods can be implemented in a rather straightforward manner.

The whole implementation for the TreeViewAdv control accessibility support is only about 300 lines of code and a few hours of coding time, but it makes a huge difference in terms of automation.

Here are two screenshots of Ranorex Spy analyzing the newly accessible tree control:
tv_spy1
tv_spy2

As you can see from the highlighted area on the first screenshot, it is now possible to look “inside” the control. This helps a lot when performing automated data entry, content validation, behavior checking, screen scraping, etc.

You can grab the source package with the accessibility addon for TreeViewAdv and the simple demo app here:

Source package for TreeViewAdv (svn rev 76) with accessibility
TreeViewAdv demo binaries

More information about the Windows Automation APIs can be found here.

Share

Tags: , ,

4 comments

  1. Frank

    Does Ranorex work also with Microsoft UI Automation?

  2. mgissing

    Yes. Ranorex uses UIAutomation for WPF and Silverlight applications.

  3. Deepak Mittal

    Can we use ranorex for automation testing of VB6.0 desktop application? If yes, how can we track third party controls like TrueDb Grid, Sheridan controls?

  4. twalter

    Hi,

    what is the result of tracking these controls with Ranorex Spy.
    Can you send us a snapshot file of your application under test to support@ranorex.com.
    Maybe the raw text plug-in is a better approach for this scenario.
    Have a look at blog entry to get more information about the raw text plug-in: http://www.ranorex.com/blog/automation-of-legacy-applications-with-the-gdi-capture-plug-in#more-926

    Regards,

Leave a Reply