General questions regarding Image-Based Automation

Ask general questions here.
tester1000
Posts: 3
Joined: Mon Jul 01, 2019 9:41 am

General questions regarding Image-Based Automation

Post by tester1000 » Wed Jul 24, 2019 10:31 am

Hi all,

I'm really new to TA and I have a few issues regarding an automation project I'm working on. Basically, the AUT generates a lot of charts, and I'd like the script to validate that these charts are correct. As an example, I've provided a snapshot of one of the charts it generates just to give you a general idea of what it is I'm trying to validate. The most obvious method for me was to simply use an image comparison with an accuracy of about 0.98, combined with various preprocessing options to validate the correctness of the charts. However, when I change the resolution of my monitor the image comparison breaks, and I'm left wondering whether using image based comparison to validate charts is even reliable enough to use.

My AUT offers the ability to extract the underlying data from a chart to the clipboard, so I can easily validate that the data is correct. It's the visualisation of data that I want to test, and without some form of image based automation, I feel like this is impossible. Any thoughts?

In short, what is the best way to validate charts and graphs?
Attachments
chart.rxsnp
(4.86 MiB) Downloaded 3 times

User avatar
Stub
Posts: 266
Joined: Fri Jul 15, 2016 1:35 pm

Re: General questions regarding Image-Based Automation

Post by Stub » Wed Jul 24, 2019 1:24 pm

Image comparisons are very delicate and thus prone to breaking. I use them on our AUT where I want to validate appearance too. Lots of data can also be acquired by other means, so I usually try to minimise any image-based comparisons. IIRC I think I'm using a comparison accuracy of about 0.999 or maybe even 0.9999, I can't quite recall. It needs just a little slop to account for inconsequential random differences here at least.

However, I don't have the resolution issue. We test at a fixed resolution, always. We use 100% scaling. The only thing I do is ensure that the client area that I'm image comparing is always exactly the same size, run to run. When you change the resolution of your monitor, are you changing display perhaps? e.g. going from laptop screen to desktop display?

To avoid differences in runtime environments I setup a system in our tests such that the image comparisons are machine-based. This allows for differences in environmental configuration - OS, Aero, etc. I recall I had a situation where I was getting different results depending on which displays I used, based on my working location. So I ended up configuring my machine to have two 'identities' if you like. One for the normal location, and another for the other location. So my image comparisons were different per-location. Awkward, but something I've just not got around to tuning out of our tests. One of these days...

tester1000
Posts: 3
Joined: Mon Jul 01, 2019 9:41 am

Re: General questions regarding Image-Based Automation

Post by tester1000 » Wed Jul 24, 2019 1:42 pm

Thanks for your reply! When I mean change resolution, I don't mean changing screens or anything- I mean simply changing the resolution settings on the same machine.

I was considering implementing a method to force the application to always run at a certain resolution, but I'm not sure how to do this with Ranorex. I've had a look at all the overloads for the RunApplication method on the API, and can't seem to find an option which defines resolution for the AUT. Furthermore, what exactly do you mean by "machine based"? Do you mean that you use a different reference image depending on which machine the test is running on? Sorry if these questions seem silly, I am very very new to Ranorex!

User avatar
Stub
Posts: 266
Joined: Fri Jul 15, 2016 1:35 pm

Re: General questions regarding Image-Based Automation

Post by Stub » Wed Jul 24, 2019 3:14 pm

We just don't change resolution, which avoids that issue. I guess you'd have to automate the Windows Control Panel to change that? Or some such similar technique.

By "machine-based" I did indeed mean per-machine images for comparison purposes. Not ideal but seemed suitable at the time and I have never gone back to look into it again. I set that up when I was new to Ranorex too.