Ranorex Consolidated Test Report

Ask general questions here.
ranoman
Posts: 26
Joined: Mon Jul 26, 2010 6:59 pm

Ranorex Consolidated Test Report

Post by ranoman » Tue Aug 24, 2010 2:48 am

I have one ranorex solution that contains about 100 test cases. I am able to generate individual reports for each of these test cases that details each step and gives a pass/fail result for each step. Below is the pseudo-code on how I accomplish this. I have the below execute method in all my test cases

Code: Select all

public void Execute()
{
   try
   {
      Report.Setup(ReportLevel.Info, "A001_NewCustomer.rxlog",true);		
      Report.SystemSummary();
      Automation_Library.Recordings.A001_NewCustomer.Start();	
      Report.End();
  }

  catch(Exception e)
  {
      Report.Failure("Test Case Failed");
      Report.Failure(e.ToString());
      Report.Screenshot();
      Modules.Instance.CloseAll();
  }
}
But here is my problem. I want to generate one master report that just contains just information of whether the test case has passed or failed with no details whatsoever. The idea being, after a round of execution is complete I would look up this master report to see what has passed and what has failed and then dig into the detailed reports of the failed test cases to see what has gone wrong. I tried a few options so far and have remained unsuccessful. Any examples on how to accomplish this is greatly appreciated.

Thanks!

User avatar
artur_gadomski
Posts: 207
Joined: Mon Jul 19, 2010 6:55 am
Location: Copenhagen, Denmark
Contact:

Re: Ranorex Consolidated Test Report

Post by artur_gadomski » Tue Aug 24, 2010 6:49 am

I am looking into something simmilar. I don't have 100 tests but I would like to get a summary, possibly in form of a web page with links to reports.
My plan is to save result of each of tests and after they're all done (or exeption happens) to create a mster report. In my case it would be a webpage but you should be able to use report system also.

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Ranorex Consolidated Test Report

Post by Ciege » Tue Aug 24, 2010 3:21 pm

You could try using the small C# app I created to this.
See this post: http://www.ranorex.com/forum/anyone-wan ... t1453.html

Your reports need to be .XML for the app to find them.

Please let me know if you have issues.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

ranoman
Posts: 26
Joined: Mon Jul 26, 2010 6:59 pm

Re: Ranorex Consolidated Test Report

Post by ranoman » Wed Aug 25, 2010 11:54 pm

Ciege,
I tried using your tool. It does not work. This is what I did:

[*]I set the test results folder path and the test plan folder path in the config file
[*]I copy pasted where my test results are stored by default and pasted them in the folder that I specified above. And yes, they are in XML format.
[*]I opened up your tool and said Refresh.

Nothing shows up.

Is there anything I am missing?

Naveen

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Ranorex Consolidated Test Report

Post by Ciege » Thu Aug 26, 2010 12:07 am

Strange, when you open the app and click tools -> config what are the settings listed?

The path to the regression test results should be a valid path to the top level folder that holds test results.

The path to test plan folder (not required) is a valid path to the top level folder that contains all your test cases. This is only used when clicking the Open Test Plan Folder button on the main dialog.

I would suggest deleting the config file that you modified and let the app create it for you with your settings.

You can also try just moving one log file to say C:\temp and point the config to C:\temp and see if it works that way.

*) After hitting refresh do you get any error or any text at all in the main screen? Like "Cannot find directory" or "QA Automation Regression Test Results" with nothing under it?
*) Make sure your Ranorex result logs are .XML extensions and not the default Ranorex Log extension.
*) Make sure you have access to the path.

I will upload the latest version of the app to the original thread in a moment.
Attachments
ConfigDialog.jpg
ConfigDialog.jpg (18.77 KiB) Viewed 1339 times
Last edited by Ciege on Thu Aug 26, 2010 12:45 am, edited 1 time in total.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Ranorex Consolidated Test Report

Post by Ciege » Thu Aug 26, 2010 12:15 am

Also, can you either post or PM me a screen shot of the main app screen and your config screen?
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

ranoman
Posts: 26
Joined: Mon Jul 26, 2010 6:59 pm

Re: Ranorex Consolidated Test Report

Post by ranoman » Fri Aug 27, 2010 7:11 pm

Hi Ciege,
Sorry for the delayed reply. Below are the answers to your questions:

) After hitting refresh do you get any error or any text at all in the main screen? Like "Cannot find directory" or "QA Automation Regression Test Results" with nothing under it?
I see "QA Automation Regression Test Results" with nothing under it. No error messages per se

*) Make sure your Ranorex result logs are .XML extensions and not the default Ranorex Log extension.
I tried with my ranorex test result being XML extensions and rxlogs. Still the same blanks results

*) Make sure you have access to the path.
It is in my desktop and I have the necessary accesses.

I will PM you the screenshots in a moment.

Thank!

ranoman
Posts: 26
Joined: Mon Jul 26, 2010 6:59 pm

Re: Ranorex Consolidated Test Report

Post by ranoman » Fri Aug 27, 2010 7:22 pm

Ciege,
I have sent the files to you.

Thanks!

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Ranorex Consolidated Test Report

Post by Ciege » Fri Aug 27, 2010 8:59 pm

PM replied. Let me know if that works for you.

Thanks!
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Ranorex Consolidated Test Report

Post by Ciege » Fri Aug 27, 2010 9:59 pm

OK, I made the fix and posted the new version.

The issue was that you had the XML file in the root folder. The app expected there to be subfolders for each test case in the root folder so it never found the XML and nothing got added to the list.
So now, if there are no subfolders in the root folder it will then look for files in the root folder.

Thanks for the heads up on this issue.

Please let me know if my fix worked for you!
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

ranoman
Posts: 26
Joined: Mon Jul 26, 2010 6:59 pm

Re: Ranorex Consolidated Test Report

Post by ranoman » Sat Aug 28, 2010 11:52 pm

Thanks Ciege! This works now!!

Just another quick question. Right now, the version tested shows up as empty for me. Is there a setting that I can use to specify the version that I am testing on?

Please let me know.

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Ranorex Consolidated Test Report

Post by Ciege » Mon Aug 30, 2010 4:18 pm

ranoman wrote: Right now, the version tested shows up as empty for me. Is there a setting that I can use to specify the version that I am testing on?
Great, glad it's working for you.

As for the version tested... We use a specific string generated by a method that is specific to our AUT in our test results that lists the AUTs version under test. This app looks for that specific string within the first few lines of the report. So, in actuality, the way it stands now you would not have anything there.

I could extend the app to allow you to configure the string to look for so long as you put that into your report near the beginning.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

User avatar
Ciege
Ranorex Guru
Posts: 1335
Joined: Thu Oct 16, 2008 6:46 pm
Location: Arizona, USA

Re: Ranorex Consolidated Test Report

Post by Ciege » Tue Aug 31, 2010 12:32 am

I've updated the app to v0.9 which adds the ability to choose your own version tested string to look for and display from your logs.
If this or any response has helped you, please reply to the thread stating that it worked so other people with a similar issue will know how you fixed your issue!

Ciege...

ranoman
Posts: 26
Joined: Mon Jul 26, 2010 6:59 pm

Re: Ranorex Consolidated Test Report

Post by ranoman » Tue Aug 31, 2010 1:41 am

Thanks Ciege!!! This is just wonderful :)

atom
Posts: 357
Joined: Sun Dec 07, 2008 11:14 pm
Location: Dublin, Ireland

Re: Ranorex Consolidated Test Report

Post by atom » Tue Aug 31, 2010 5:06 pm

Hiya

Another simple way to get full reports is just to save some information for each test case run to a database
(Oracle/SqlServer/Sqlite for example)

e.g.
Have a database table like:

Field Type
RunId Number
Version VarChar
TestName VarChar
Result VarChar

For each test add a little bit of code to INSERT a record into this table

Now to get the report you have many options, there are many tools that create reports from a database
Perhaps one of the easiest is to just use Excel, with an ODBC connection to your results database, and using an SQL query like:

SELECT TestName, Version FROM TABLE WHERE Result='Fail'

or another example:

SELECT Version, Result, Count(*) FROM TABLE GROUP BY Version, Result