screen shots compared incorrectly

[jyothsna]

HI,
Im currently using the trial version of Ranorex 3.3 studio.
The application that I want to record the tests is such that the User Interface is a desktop application while the visualization is on the web browser.
The web browser displays images and we can use the buttons on the UI to do any manipulation to the images like rotate, draw line etc.
I created a 2x2 layout on my browser to display 4 unique images ,while recording the test , I clicked on validation in the record module and then chose compare images options ,
a correct screen shot was captured and the step in the record module showed the correct screenshot.

I noticed that in the play back mode one of the 4 images in the web browser was different. as compared to screenshot(this is because of a bug in the application)
however my test still passed , and the report showed the 2 wrong screenshots it compared and still has marked it pass, I expected the test to fail and capture the defect.

I also played around with similarity settings in imaging. I noticed that when I keep the similarity threshold as 1, then my test fails for correct screenshots.
and when I keep the threshold any value less than 1 my test passes for all the screenshots including the wrong one.

could you please let me know what I have missed here as I could not find any useful information in the help guide to solve this issue.

Thanks
Jyothsna

[Support Team]

Hello,

Could you please provide us screenshots of these images in order to reproduce this issue?
You could try to compare only the region which varies.

Which similarity settings have you tried so far?

Regards,
Markus (T)

[nayara.crema]

Hello,
I'm using the Ranorex 3.3.4, and I'm in the same situation reported by jyothsna. I used the options "ContainsImage" and "CompareImage" but realized that the image is not compared with any of these options, it seems he does not compare the contents of the image, but its coordinates.
I also had a situation where he hoped that the test would fail because of differences in the expected image with the image obtained, however the execution was successful. Would a return of support personnel Ranorex to clarify how the comparison pictures really works. I thank you.

[Support Team]

Would a return of support personnel Ranorex to clarify how the comparison pictures really works

For both ContainsImage and CompareImage a screenshot of the associated element is made at runtime and compared to the image stored in the action.
ContainsImage checks whether the specified image can be found somewhere inside the screenshot.
CompareImage compares the image with the screenshot, i.e. both images need to have the same size.

Similarity: Defines (as percentage) how similar the compared images need to be in order to pass the check. The color differences for each pixel is calculated and the mean squared differences are summed up.
Example: Imagine that we compare 10x10 pixel color images. If all pixels have the same color except for one pixel being white (RGB 255,255,255) in picture A and black (RGB 0,0,0) in picture B, then the similarity is 99%.
If all pixels have the same color except for one pixel being black in pic A and one being gray (RGB 128,128,128, i.e. 50% color difference) in pic B, then the similarity is 99,75% (because of the squared error).
Simply speaking, a similarity of 99% is already a quite low similarity if you compare large images and want to find small differences.

...and the step in the record module showed the correct screenshot

The image shown by the "Screenshot" tab on the right side of the recorder has nothing to do with image comparison, but only shows a screenshot taken when the action was recorded. Just to make that clear

however my test still passed , and the report showed the 2 wrong screenshots it compared and still has marked it pass, I expected the test to fail and capture the defect.

What value did you choose for the Similarity? Always keep the similarity as close to "1.0" as possible, in most cases it should be "1.0".
If the validation fails with a similarity of "1.0", then the newly taken screenshot is not exactly similar to the stored one (a slight color change, for example, is enough to make the validation fail).

it seems he does not compare the contents of the image, but its coordinates.

As explained above, "CompareImage" expects the screenshot taken from the element to have the same geometry as the stored image. "ContainsImage" searches for the stored image within the screenshot.

If you are still having problems with the image comparison, could you post some sample images that you try to compare and the settings you use?

Regards,
Alex
Ranorex Team

[nayara.crema]

Hello,

I have attached some pictures as requested.

In image 1, we have the expected image.
In image 2, the parameters have the expected image.
And in figure 3, we have the image to be obtained. If you compare the image 1 and 3 we can see a new option is shown in the submenu, so the test should fail.
However, as can be seen in Annex 4, the test is successful.

I look forward to returning.

Thank you!

[Support Team]

I had a look at the images and your CompareImage settings. The outcome is as excepted, i.e. the validation correctly succeeds, since you specified a similarity of "0.9" (=90%).

As I explained in my previous post, a similarity of 90% is very low. The two images you compare are 1000x42px in size, the button that appears is (at most) 150x20px. So that's even less than 10% of the pixels of the image.
And even if more than 10% of the pixels would change, image comparison checks for color changes. So your validation would still succeed (even if more than 10% of the pixels would change), since the color changes are smaller than 10% of all possible color changes (black to white image).

You should change the similarity to a value close to 1.0 (e.g. 0.9999) or just 1.0.

Regards,
Alex
Ranorex Team

[nayara.crema]

Hello,

Actually, I changed to 1.0 similarity and started accusing failure. But failure also accused in items where the content is the same, what changes between images is a small displacement (see images 1 and 2 attached), and this difference would not be that serious to want to validate, since the my focus is the image content, and the content is the same.
So now I have another problem regarding the comparison of images ....
For clarity, I will explain basically what is my test:
I need to validate menu items, happened to appear if new items or failed to show any of them.
All I need to do this validation in IE and Chrome browsers, then use the exact same script to test the 2 browsers.
But the expected image was recorded on IE, and when testo in chrome, the image undergoes a small change, but for my test is not serious, as I said, for me the important thing is its content.
I can validate the content by Ranorex?

Thank you.

[Support Team]

what changes between images is a small displacement

Then you should change your validation to "ContainsImage" and select only the relevant part of the image. Then the validation will succeed if the relevant part is found within the screenshot made at runtime.

And if the image is not rendered a 100% identically in IE and Chrome, just change the similarity to 0.9999 or similar (the exact value needs to be either calculated or iteratively evaluated by you).

Regards,
Alex
Ranorex Team

[nayara.crema]

Hello,

Even selecting ContainsImage seems that is not comparing the content of the image ...
In the attached image informed similarity of 97%, and yet accuses fails in comparison, and the content of the images is the same, what changes is expected that the image was captured in the IE browser, but is obtained from nagevador Chrome.

In other pictures, I had to inform a particular value for the similarity so that compared to successfully perform in Chrome, however what worries me is that this figure may be low when the test was run in IE, then it will not acknowledge failure, if it exists, it is not as near to 100% similarity (values ​​used in other images 0.982, 0.985, 0.998, 0.991).


Thank you.

[Support Team]

Hello,

Have you already tried to use 'Find Region' option in the 'Image Editor' to locate the exact image region?
More info can be found in our User Guide.

Is it possible to identify your menu items in Spy?
If yes, you could get these items by iterating elements within a parent element (e.g. main menu item).

Could you please post or send a Snapshot of your application?
Please take a look at section 'Creating Ranorex Snapshot Files' from our User Guide.

Regards,
Markus (T)