Anyone Want an Automation Results Dashboard Utility?

Experiences, small talk, and other automation gossip.
User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Wed Jun 23, 2010 10:39 pm

I have been creating a dashboard utility for displaying automation test results. It is an ongoing process but pretty useful for us here.
Was wondering if anyone would want a copy of something like this for their use at their place of work?
If there is any interest I could probably get something out to the group by late this week or early next week after I make a couple of changes and upgrades.

Please let me know!

Functionality:
*) Will display all test results found in a convenient treeview.
*) Will allow opening the log file directly in IE.
*) Will highlight tests that have been executed within the last X number of days.
*) Optional - Will display Red/Yellow/Green for tests that have Failed/Warning/Passed.
*) Optional - Will bubble up the test results to the parent node(s).

Requirements:
*) All test results must be stored in a central location accessible by the local user (in my case all my test results are stored on a network share accessible to everyone).
*) Currently only logs converted to XML are supported.


See below for a screenshot of the current rev of the utility:
v13.jpg
Resgression Results Dashboard
v13.jpg (94.83 KiB) Viewed 5778 times
Last edited by Ciege on Wed Feb 02, 2011 12:31 am, edited 1 time in total.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Fri Jun 25, 2010 6:04 pm

OK, here it is. If interested, please give it a shot and let me know what you think.
Just extract the file and run.

v0.6 - Adds ability to find RXLOG files as well as XML files.
v0.7 - Better information displayed for access denied folders.
v0.8 - Fix if no subfolders in root folder.
v0.9 - Added config for Version Tested string in log files.
v0.10 - Added basic Report generation.
v0.11 - Added ability to export .TXT or .XML reports.
v0.12 - Added ignore OLD and RETIRED folder option to allow utility to skip old or retired folders
v0.13 - Fix for ordering test result files. Now in last write time order.
v0.14 - Added option for how to get test results when requested.
v0.15 - Added support for case insensitive log level results.
Attachments
DisplayAutomationLogs0.15a.zip
(20.56 KiB) Downloaded 520 times
Last edited by Ciege on Fri Feb 03, 2012 6:54 pm, edited 12 times in total.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

atom
Posts: 357
Joined: Sun Dec 07, 2008 11:14 pm
Location: Dublin, Ireland

Re: Anyone Want an Automation Results Dashboard Utility?

Post by atom » Thu Jul 01, 2010 4:28 pm

Hiya

Hmm.... Nice idea, but its been done before

If you use visual studio say, develop your test cases using MSTest
You then already get a results dashboard

If you develop your tests using NUnit, the NUnit GUI Runner will display all results to you, and produce an xml file of all your results

If you code direct .exe's then perhaps do what we do:
- create a TestResult class
- each test case populates a TestResult class, and at the end calls a Save method
- Save, method writes the result to oracle/sql server/access, closes the ranorex log file, and copies it to a shared directory
- Develop a small web application to load results from Oracle and display then in a web page
with a link to the ranorex log file



Cheers

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Thu Jul 01, 2010 5:13 pm

Certainly good ideas. However, what my task is here (as the only automation person) is to check the results of a test myself to verify the results are accurate and not a bug in my automation code that caused an issue or a timeout issue, etc... Only then, after I have verified the results (as a true pass or fail) do I move the results XML file(s) to the shared directory of results for the rest of the QA team and development team to see.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

atom
Posts: 357
Joined: Sun Dec 07, 2008 11:14 pm
Location: Dublin, Ireland

Re: Anyone Want an Automation Results Dashboard Utility?

Post by atom » Thu Jul 01, 2010 5:48 pm

Not one for blowing my own trumpet but I did have an idea to sell such a product:

http://www.testresultx.com/

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Thu Jul 01, 2010 5:53 pm

Uh-Oh, did I step on your toes by giving mine away for free?
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

atom
Posts: 357
Joined: Sun Dec 07, 2008 11:14 pm
Location: Dublin, Ireland

Re: Anyone Want an Automation Results Dashboard Utility?

Post by atom » Thu Jul 01, 2010 6:45 pm

Ahh no didnt actually start writing that anyway!

Was just an idea... as it seems most major automation players (except microsoft), forget that the most important thing about automation is the results, this is where the value of automation can be seen

If you have to scan 100's/1000's of log file to know if all tests pass its a pain
To compare current run of of tests, with previous run to see improvement is again quite painful when all you have is log files

Test managers dont care about ranorex log files to be honest
What they want to see is a gobal picture:
- how many test ran
- how many passed / failed
- is it better / worse than previous run
- what is the over-all coverage
- etc.

So to create a global picture, having all results, of all runs stored in one place, in a convienient format (Oracle, SQL Server, Access say), makes this very easy!

Anyway that was my aim with that, never actually got time to do it

atom
Posts: 357
Joined: Sun Dec 07, 2008 11:14 pm
Location: Dublin, Ireland

Re: Anyone Want an Automation Results Dashboard Utility?

Post by atom » Thu Jul 01, 2010 6:56 pm

Sorry im not being critical of your tool, i think its good to share such stuff!

I just think Ranorex should do something to define what is a result, and have a way to store it, rather than us all developing our own ways

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Thu Jul 01, 2010 7:09 pm

No, I didn't take it that way at all... So no worries there...

However, I do agree about Ranorex providing a results dashboard thing as well. But I'm more than happy to have them deliver the strongest test solution they can at an affordable and competitive price before they delve into the results half of it. I know they are a pretty small team compares to the "big guys" and certain things need to take priority. So if I (or anyone) can provide a temporary, albeit un-polished, solution to the reporting aspect before Ranorex has time and manpower to do it I'm fine with that.

I see that my utility has been downloaded 5 times thus far but no comments on it. I just want people to give it a shot and let me know how it works out for them and if there are any additional features they would like added to it until an "official" utility is released by Ranorex. I'm more than willing to spend my copious amounts of extra time :roll: on it to make it better.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

JayDee
Posts: 20
Joined: Wed May 26, 2010 8:53 am

Re: Anyone Want an Automation Results Dashboard Utility?

Post by JayDee » Thu Jul 08, 2010 10:28 am

Hey Ciege,

i want to try your tool but it don't detect any of my logfiles. I converted them into XML and set the paths... any idea what i did wrong?

Greetings, jd

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Thu Jul 08, 2010 3:56 pm

After you set the paths in Config did you click the refresh button?
Do you get a message in the app that it cannot find the directory and to set it in Tools -> Config?
Does your machine have access to the path you set?
Is the path correct in the Config.XML file that gets created?
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Thu Aug 26, 2010 12:10 am

Latest version 0.5 has been added to the second post.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Fri Aug 27, 2010 4:49 pm

Updated to 0.7.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Wed Sep 01, 2010 11:16 pm

Updated to v0.10.
Added some fixes and a basic report generation ability.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Anyone Want an Automation Results Dashboard Utility?

Post by Ciege » Mon Sep 20, 2010 5:49 pm

Updated to v0.11.
Added ability to export .TXT or .XML reports.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...