Ranorex Help Center

How can we help you?

Image-based Automation

In some situations, required information for automation is not available as text but only as pixel image. Ranorex provides image-based automation to bypass this lack of accessibility.

Screencast Information about image based automation can also be found in our screencast "How and When to Use Imaging Capabilities". Please follow the link to view this video: http://youtu.be/wwr5ddOe4h0


Sometimes you need to automate a click action based on image information. For this reason, Ranorex Recorder provides an optional way to capture image related mouse actions. To activate image-based recording, simply check the 'Image based' checkbox in the Recorder's toolbar during a recording session.

'Image based' recording activated

'Image based' recording activated

Now move the mouse pointer over a certain UI element. Ranorex highlights the UI element below the mouse and also highlights a recognized image region within the UI element. To turn off image-based recording, uncheck the checkbox or press the shortcut key 'I'.


Note Hotkey functionality has to be activated by first using the master hot key 'SCROLL'. Read more about hotkey usage in the current lesson in the section entitled Recorder Hotkeys.

Introductive Example

In KeePass, there is a way to include some user specific icons for password entries (e.g. someone could use the WordPress logo for his WordPress password entry). For automation of a click on a custom icon, it might be necessary to use image-based automation because the order and in addition the index of the icon can change because icons with a lower index can be deleted.
KeePass Icon Picker dialog containing two custom icons

KeePass Icon Picker dialog containing two custom icons

To prepare image-based automation for a click on a custom icon, activate image-based recording. Move the mouse pointer to the listview for custom icons in KeePass and wait until the whole listview frame is highlighted. Execute a mouse click on the listview (take care to not click on a custom icon). The click has to be done on the listview control in contrast to a click on an (icon) element itself because this click defines the region in which the prospective image has to searched. Thus the image is searched in the entire list control.

After stopping the recording, the actions table contains an image-based mouse click action. At the time of recording the mouse click, a screenshot of the control was created and is now listed as a child element of the corresponding repository item.


The Ranorex Studio Image Editor

The Ranorex Studio Image Editor

In order to automate the image-based click on the custom icon, you have to tell Ranorex which image (or part of the image in this case) you want to find and click within the figure of the listview. Open the image editor by clicking on the button in the location cell (see graphic above). In this dialog use the 'Select image find region' toolbar button to define the user-defined icon which should be clicked. Additionally use the 'Select click location' functionality to tell Ranorex at which position the mouse click should be executed. This features will be described in more detail later on in the   Image Editor section.
'Select image find region' and 'Select click location' in Ranorex Image Editor

'Select image find region' and 'Select click location' in Ranorex Image Editor

In this example, Ranorex has been used to automate a mouse click on the custom WordPress logo within the entire listview of all available custom icons. If found, this icon is clicked regardless of the position of the icon. Again, Ranorex is not searching for any index or any list item at a specific position but for a specific figure within a region (which is a screenshot of the listview in this case). To check it, delete the KeePass icon with the index 0 and execute the recording again. The WordPress icon will be found and will be clicked although the position and the index changed in comparison to the time of recording.

To alter the settings of an image-based automation action, open the property group 'Image-Based Location' in the property grid.

Image-based Location Settings

Advanced Options Use the property group to set advanced image search options like 'Clipping' or 'Best Match'

If 'Best Match' is set to true, the result position with the highest similarity is used for validation. If is set to false, the first available result position will be used for the validation. The first is more accurate and the second is faster.
Preprocessing Defines preprocessing steps that can be performed on an image before search (see Image Preprocessing Filters below).
Screenshot Name Specifies the name of the screenshot used in the search.
Selection Rectangle Defines the image selection region (=  what to search for).
Similarity

Specifies the minimum similarity (0.0-1.0), that the image region to search for needs to have in common with the image in order to be considered a match (for more information, see Similarity below)


Image Preprocessing Filters

None: No preprocessing

Ranorex logo with no preprocessing

Grayscale: Convert the image to a grayscale image.

Ranorex logo in grayscale

Edges: Detect object edges in the image using the Laplace edge detection algorithm (includes Grayscale).

Ranorex logo with Laplace edge detection

EdgesSobel: Detect object edges in the image using the Sobel edge detection algorithm (includes Edges).

Ranorex logo with Sobel edge detection

Downsize: Downsize the image.

Downiszed Ranorex logo

Threshold: Convert the image to a black/white image using a threshold (includes Grayscale).

Thresholded Ranorex logo

Similarity

The similarity value can be adjusted from 0.0 to 1.0. This corresponds to 0 % similarity (completely different pictures) and 100 % similarity (completely identical pictures). It may be tempting to use values like 0.8 or 0.9 to ensure the image is found even if some superficial changes occur. However, these values are only seemingly high. In reality, they are actually very low already.

At 0.9 similarity, an entirely white 100-pixel picture would be considered identical to a picture with 90 white and 10 black pixels. That’s quite a difference already. When you start comparing images in the magnitude of several thousand pixels, the optical deviations can be even more striking.

Consider the icons of Edge and Internet Explorer. They are each around 2000 pixels and markedly different from each other. A 0.9 value would not catch these differences. It would consider them a match. In fact, you would need a minimum value of 0.95 for them to be treated as different.

 

Identical pictures at a similarity value of 0.9


For this reason, we recommend you use a similarity value of 1.0 or 0.9999. To ensure your images are found at these high values, make sure to use uncompressed image formats, such as .png and .bmp. The artifacts created during compression make formats like .jpg unsuitable.

For large pictures in the order of several thousand pixels and more, we also recommend you turn off similarity reporting, as it can take a very long time to compute even on fast machines.


Image-Based Settings

To set up the default values for image-based recording, open up the 'Settings' dialog and continue with activating the 'Imaging' tab.

Settings dialog for image-based recording

Settings dialog for image-based recording

Image Editor

Use the context menu item 'Edit Location...' to open the image editor and to change the click location options.

The editor provides a more detailed view of the captured image information. In addition, it offers some useful features for changing the image rectangle to be searched for.


Click location dialog to change the settings of image-based mouse actions

Click location dialog to change the settings of image-based mouse actions

Image Editor Functionality

Image Based Specifies whether the click is based on image information or not

Select Image Find Region

Select Image Find Region Specifies the region to search within the image

Select Click Location

Select Click Location Defines the click position in relation to the searched region

Autoselect Image Find Region

Autoselect Image Find Region Helps to specify the search region within the image

Zoom Out

Zoom Out

Zooms out

Zoom 100% Switches to 1:1 view

Zoom In

Zoom In Zooms in

Capture New Screenshot

Capture New Screenshot Captures a new screenshot from the related repository item

Use the drop-down combo boxes to specify whether a mouse action should be performed relative or absolute to the currently defined 'Image Find Region'.

Absolute location means the distance from the upper left corner of the 'Image Find Region' to the click location (in pixels).

Relative location offers a couple of pre-defined location settings (e.g. Center, CenterLeft ...) relative to the 'Image Find Region'. It's also possible to define relative position statements for the click location. Therefore the click-location inside the 'Image Find Region' is defined by a value between 0 and 1 for both directions (X and Y-Axis). A relative click location of "0.9;0.9" defines a click near the bottom right corner of the specified region. Values greater than 1 (or negative values) mean locations outside the defined region relative to the size of the region. 


Set relative or absolute mouse action location

Set relative or absolute mouse action location

If the captured screenshot has to be updated, click the 'Capture New Screenshot' toolbar button. Every newly created screenshot is also automatically added to the image list of the related repository item so it's available for other image-based validation (checkpoint) or automation actions.

Add a new image or select an existing image to specify the location/graphic to search

Replace expected with actual image

This function allows you to replace the image that Ranorex Studio expected to find with the image that it actually found directly in the report opened in Ranorex Studio. This simplifies the process and in many cases makes it easier to fix errors related to the Contains Image or Compare Image validation actions.

To replace an expected image, follow these steps:

  1. In the report opened in Ranorex Studio, find the failure message showing the actual image.
  2. Mouse over it and click ‘Replace expected image’ in the top-right.
    Replace image button
  3. Confirm the dialog that appears.
  4. Make any adjustments you think are necessary and confirm the Edit Image Validation dialog to complete the process.

The validation will now look for the new image. The original image won’t be deleted and can still be viewed as part of the repository item.