Ranorex Consolidated Test Report : General Questions

[ranoman]

I have one ranorex solution that contains about 100 test cases. I am able to generate individual reports for each of these test cases that details each step and gives a pass/fail result for each step. Below is the pseudo-code on how I accomplish this. I have the below execute method in all my test cases

Code: Select all

public void Execute()
{
   try
   {
      Report.Setup(ReportLevel.Info, "A001_NewCustomer.rxlog",true);      
      Report.SystemSummary();
      Automation_Library.Recordings.A001_NewCustomer.Start();   
      Report.End();
  }

  catch(Exception e)
  {
      Report.Failure("Test Case Failed");
      Report.Failure(e.ToString());
      Report.Screenshot();
      Modules.Instance.CloseAll();
  }
}


But here is my problem. I want to generate one master report that just contains just information of whether the test case has passed or failed with no details whatsoever. The idea being, after a round of execution is complete I would look up this master report to see what has passed and what has failed and then dig into the detailed reports of the failed test cases to see what has gone wrong. I tried a few options so far and have remained unsuccessful. Any examples on how to accomplish this is greatly appreciated.

Thanks!

[artur_gadomski]

I am looking into something simmilar. I don't have 100 tests but I would like to get a summary, possibly in form of a web page with links to reports.
My plan is to save result of each of tests and after they're all done (or exeption happens) to create a mster report. In my case it would be a webpage but you should be able to use report system also.

[Ciege]

You could try using the small C# app I created to this.
See this post: http://www.ranorex.com/forum/anyone-want-an-automation-results-dashboard-utility-t1453.html

Your reports need to be .XML for the app to find them.

Please let me know if you have issues.

[ranoman]

Ciege,
I tried using your tool. It does not work. This is what I did:

[*]I set the test results folder path and the test plan folder path in the config file
[*]I copy pasted where my test results are stored by default and pasted them in the folder that I specified above. And yes, they are in XML format.
[*]I opened up your tool and said Refresh.

Nothing shows up.

Is there anything I am missing?

Naveen

[Ciege]

Strange, when you open the app and click tools -> config what are the settings listed?

The path to the regression test results should be a valid path to the top level folder that holds test results.

The path to test plan folder (not required) is a valid path to the top level folder that contains all your test cases. This is only used when clicking the Open Test Plan Folder button on the main dialog.

I would suggest deleting the config file that you modified and let the app create it for you with your settings.

You can also try just moving one log file to say C:\temp and point the config to C:\temp and see if it works that way.

*) After hitting refresh do you get any error or any text at all in the main screen? Like "Cannot find directory" or "QA Automation Regression Test Results" with nothing under it?
*) Make sure your Ranorex result logs are .XML extensions and not the default Ranorex Log extension.
*) Make sure you have access to the path.

I will upload the latest version of the app to the original thread in a moment.

[Ciege]

Also, can you either post or PM me a screen shot of the main app screen and your config screen?

[ranoman]

Hi Ciege,
Sorry for the delayed reply. Below are the answers to your questions:

) After hitting refresh do you get any error or any text at all in the main screen? Like "Cannot find directory" or "QA Automation Regression Test Results" with nothing under it?
I see "QA Automation Regression Test Results" with nothing under it. No error messages per se

*) Make sure your Ranorex result logs are .XML extensions and not the default Ranorex Log extension.
I tried with my ranorex test result being XML extensions and rxlogs. Still the same blanks results

*) Make sure you have access to the path.
It is in my desktop and I have the necessary accesses.

I will PM you the screenshots in a moment.

Thank!

[ranoman]

Ciege,
I have sent the files to you.

Thanks!

[Ciege]

PM replied. Let me know if that works for you.

Thanks!

[Ciege]

OK, I made the fix and posted the new version.

The issue was that you had the XML file in the root folder. The app expected there to be subfolders for each test case in the root folder so it never found the XML and nothing got added to the list.
So now, if there are no subfolders in the root folder it will then look for files in the root folder.

Thanks for the heads up on this issue.

Please let me know if my fix worked for you!

[ranoman]

Thanks Ciege! This works now!!

Just another quick question. Right now, the version tested shows up as empty for me. Is there a setting that I can use to specify the version that I am testing on?

Please let me know.

[Ciege]

Right now, the version tested shows up as empty for me. Is there a setting that I can use to specify the version that I am testing on?

Great, glad it's working for you.

As for the version tested... We use a specific string generated by a method that is specific to our AUT in our test results that lists the AUTs version under test. This app looks for that specific string within the first few lines of the report. So, in actuality, the way it stands now you would not have anything there.

I could extend the app to allow you to configure the string to look for so long as you put that into your report near the beginning.

[Ciege]

I've updated the app to v0.9 which adds the ability to choose your own version tested string to look for and display from your logs.

[ranoman]

Thanks Ciege!!! This is just wonderful

[atom]

Hiya

Another simple way to get full reports is just to save some information for each test case run to a database
(Oracle/SqlServer/Sqlite for example)

e.g.
Have a database table like:

Field Type
RunId Number
Version VarChar
TestName VarChar
Result VarChar

For each test add a little bit of code to INSERT a record into this table

Now to get the report you have many options, there are many tools that create reports from a database
Perhaps one of the easiest is to just use Excel, with an ODBC connection to your results database, and using an SQL query like:

SELECT TestName, Version FROM TABLE WHERE Result='Fail'

or another example:

SELECT Version, Result, Count(*) FROM TABLE GROUP BY Version, Result