Image recognition not working, always matches over 95%

Class library usage, coding and language questions.
aaroncz
Posts: 5
Joined: Tue Jun 28, 2016 3:22 pm

Image recognition not working, always matches over 95%

Post by aaroncz » Tue Jul 05, 2016 5:08 pm

Hey,

I'm having some problems with image recognition. What I would like to do is have a screenshot of a map (repository item named canvas), and i have specific places on this map (screenshots of the map in various locations), and I would like to validate that they appear within a certain amount of time (5 seconds) when the map goes to those locations.

The problem at I am coming across is that it will almost always validate the images no matter where they are or what they look like. I had the similarity at 0.98 before which prevented most of the incorrect validations but also didn't correctly validate sometimes, but just about everywhere on the map will be successful at 0.95. I added EdgesSobel hoping this would solve the problem as the edges should be very different but it will still work.

I've tried using 'Validate.ContainsImage()' and I can't seem to get anywhere with that. I also made a screenshot of the map at real time, resized it to be bigger and then tried 'Imaging.contains()' but again no luck there. They both have very high similarity rates even when the images don't look alike at all.

Main questions:
1-I was trying to use 'Imaging.Find()' to find where the similarities were but I'm unsure how to post the results of this to the report and check where all the similarities are in the images. Is there a way to do this?

2-How does the imaging validation.contains work, pixel by pixel or if there is just a similar amount of pixels contained anywhere in the images?

3-I'm also wondering if someone can point me in the right direction for my image validation. I've attached a zip containing the current images I'm working with. I'm hoping the 'image_to_check_fail' and 'image_to_validate ' should have a low similarity as the image doesnt exist in it, where as 'image_to_check_successful' should have a very high similarity as it is contained in it.

I've attached a zip containing 3 images:
image_to_check_successful (matching against this should be successful)
image_to_check_fail (matching against this should fail)
image_to_validate (this is the image which should be found)
Attachments
image files.zip
Images used to validate
(159.86 KiB) Downloaded 33 times

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Image recognition not working, always matches over 95%

Post by Support Team » Tue Jul 05, 2016 7:26 pm

Hi aaroncz,

The meaning of the similarity value is best explained in that post:
http://www.ranorex.com/forum/screen-sho ... tml#p17565

In short: 95% is a very very low similarity when comparing color images.

A few hints to get your image comparisons going:
  • Do not use lossy image formats like JPEG, but use lossless compression formats like PNG (or uncompressed BMP). Only that way you can achieve a similarity of 1.0. (In your example, the "image_to_validate.jpg" only has a similarity of 0.999889 within the "image_to_check_successful.jpg".)
  • If comparing large images, always try to achieve a successful similarity of 1.0; otherwise the comparison will take very long (can be minutes).
  • Do not use any preprocessing options unless absolutely necessary, because it will (almost always) lower the achievable similarity.
I was able to successfully find the "image_to_validate.jpg" with a similarity of 0.999 in the "image_to_check_successful.jpg" and not find it in the "image_to_check_fail.jpg" using the same similarity.
That said, with PNG images you may even use a similarity of 1, which would speed up the search tremendously.

Regards,
Alex
Ranorex Team
.
Image

aaroncz
Posts: 5
Joined: Tue Jun 28, 2016 3:22 pm

Re: Image recognition not working, always matches over 95%

Post by aaroncz » Wed Jul 06, 2016 11:24 am

Hi Alex,

Thanks for the hints, I'll raise the similarity to 0.99 or above and not use preprocessing.

Setup for tests:
-Recording containing 1 'Validate.ContainsImage'. New screenshot of app with maximized window (app is a map display), and using this to validate against
-Similarity set to 0.999, no preprocessing, timeout 5s
-Selected a boarder area of about 85% of the screenshot to validate against

Tests(validate image at different zoom levels on map):
-Just a straight match against the exact image, results 0.9999 similarity but took 4mins 34secs

-Zoomed out, result 0.996 (failed but very close so counting this as okay), took 4mins 36secs

-Lowered similarity to 0.99. Zoomed out again, match at 0.994, took 4mins 34secs

-Changed map to different location, failed (expected) match at 0.94 similarity, took 4mins 37secs


Question:
This is much better but what I would like is for the timeout to work. Is teh 'Timeout' meant to timeout the validation after a certain amount of time? I'm adding items to the map and would like to validate that the item appears after X amount of time. Is this possible?

I was using user code, passing in a int 'WaitTime' variable to the 'Imaging.FindOptions()' with an amount of time I want the validate to wait, but no matter what I do it always seems to just continue and go over that time.

So it doesn't seem to work in either the recording or by being set in user code.

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Image recognition not working, always matches over 95%

Post by Support Team » Fri Jul 08, 2016 2:43 pm

Hello aaroncz,

The timeout specifies the time to iteratively search for the specified feature, in your case the image. May I ask what exactly you mean with:
but no matter what I do it always seems to just continue and go over that time.
Do you want to wait, before validating the image? In that case I suggest adding a delay action before validating the image.
Would you mind sending us a sample solution which would better help us understanding your problem.

Regards,
Bernhard
.
Image

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Image recognition not working, always matches over 95%

Post by krstcs » Fri Jul 08, 2016 2:58 pm

I think he might be referring to Ranorex's occasional issue of disregarding the search timeout when searching for elements.

I don't see it consistently, and I can't reproduce it, but occasionally when Ranorex searches for an element that isn't available, the search will not timeout. It just continues searching, eventually locking up the SUT and Ranorex test, especially on IE web pages.

It acts like the timeout timer's logic is set to = instead of >=.

Code: Select all

if (timer = timeout) break;
//instead of
if (timer >= timeout) break;
It hits most of the time, but occasionally it just skips over the end of the timeout and goes off in the weeds.
Shortcuts usually aren't...

User avatar
Support Team
Site Admin
Site Admin
Posts: 11709
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Image recognition not working, always matches over 95%

Post by Support Team » Sat Jul 09, 2016 4:18 pm

I have to split my answer in two parts to address the two completely separate issues discussed in this thread:

1. The initial image recognition issue:
aaroncz wrote:Is teh 'Timeout' meant to timeout the validation after a certain amount of time?
The documentation of the Imaging.FindOptions.Timeout property is pretty clear: The timeout will not abort a search iteration, only provide a way to continuously search for a feature within a changing image, i.e. within screenshots got from a changing element.
aaroncz, since searching your image just once already takes ~5 minutes, there's no sense in specifying a timeout of below those 5 minutes.

As I tried to explain in my first post, IMHO you'll need to rethink the whole image recognition setup. The image files you are searching are huge (in terms of pixels) and searching them without using a similarity of exactly 1 will always take very long.
So, please use PNG or BMP images, not JPG as you will not be able to use a similarity of 1 with JPG images.
Moreover, the Ranorex image recognition algorithm does not consider zooming images. So, searching in a zoomed out image makes no sense, you just get weird results. Ranorex only supports the situation where image and feature are at the same zoom level.


2. The general element wait timeout issue referred to by krstcs:
krstcs wrote:but occasionally when Ranorex searches for an element that isn't available, the search will not timeout. It just continues searching, eventually locking up the SUT and Ranorex test, especially on IE web pages.
The only situations we know this may happen is when there are native calls from Ranorex to the SUT that cannot be aborted. This is when Ranorex needs to use native interfaces to recognize the UI, namely: MSAA, UIA, Internet Explorer plugin if Ranorex addon is not installed in IE. As those interfaces do not allow to specify a timeout and use native calls, the calls cannot be aborted. When the SUT locks up, i.e. does not return the call, the Ranorex execution may be blocked.
If you can reproduce that behavior reliably, we are keen on investigating the issue and see if it is one of the known ones or another problem :)
krstcs wrote:It acts like the timeout timer's logic is set to = instead of >=.
Well, I can assure you that this timeout issue is not caused by such a simple mistake. If that would be the case, the timeout would almost never be correctly used, since the check would need to happen exactly at the right time :wink:

Regards,
Alex
Ranorex Team
.
Image

krstcs
Ranorex Guru
Posts: 2683
Joined: Tue Feb 07, 2012 4:14 pm
Location: Austin, Texas, USA

Re: Image recognition not working, always matches over 95%

Post by krstcs » Mon Jul 11, 2016 2:52 pm

Yeah, I didn't mean to imply that that was the actual logic, just that it's what the system acts like sometimes.

Just thought that may be what @aaroncz was referring too.
Shortcuts usually aren't...