Asserting on log file contents from test

[Groostav]

Hey guys,

I'm a dev at a startup and recent purchaser or ranorex. What I would like to do is get our tests to assert on the contents of our applications log files. We're building a desktop java app and thus use java.util.logging (JUL) to generate log files, which can be configured to spit out XML. What I'd like to do is add an assertion step that gets executed frequently (between every step?) that asserts that the contents of the log file do not contain any new log entries at a level greater or equal to WARNING.

A sample XML file from jul:

Code: Select all

<record>
  <date>2014-06-08T13:43:19</date>
  <millis>1402215199326</millis>
  <sequence>0</sequence>
  <logger>com.javacodegeeks.corejava.util.logging.LoggerExample</logger>
  <level>CONFIG</level>
  <class>com.javacodegeeks.corejava.util.logging.HandlerExample</class>
  <method>main</method>
  <thread>1</thread>
  <message>Configuration done.</message>
</record>
<record>
  <date>2014-06-08T13:43:19</date>
  <millis>1402215199376</millis>
  <sequence>1</sequence>
  <logger>com.javacodegeeks.corejava.util.logging.LoggerExample</logger>
  <level>FINE</level>
  <class>com.javacodegeeks.corejava.util.logging.HandlerExample</class>
  <method>main</method>
  <thread>1</thread>
  <message>Finer logged</message>
</record>

At the lowest level, our default exception handler, like most applications (I suspect), is configured to fire a 'something strange has happened!' UI element and log the exception at level SEVERE. We've actually gone a little bit above that through use of some annotations and static methods to help describe context, so we generally have pretty good error messages even when a completely unknown exception has happened. There are also other performance sensitive non-exception-flow places that simply log problems directly to the loggers as they encounter problems. I would very much like ranorex to reflect these failures.

Our applications issue as written

As a QA I would like my the Ranorex tests to be able to quickly determine if an error was logged to the log files so that I can have a test suite which is sensitive to dev-designed failure modes and publish the contents of warnings generated by [our application], as well as over failures in the UI, to the ranorex reports.

Currently, in order to have the Ranorex tests assert that the logs don't contain any warnings, [our head of QA has ranorex open the log files in notepad and asserts that the contents are empty. This is a slow, machine dependent, error prone process.

A better tool would be to have [our application] output XML encoded logs, and then run an XQuery against those logs using some XML facilities built into the .net framework --or perhaps supplied by newtonsoft, or another big serializer. This issue would involving writing custom source code in C# for use with the ranorex tests. That code would loosely be written as

Code: Select all

var logs = XMLSupport.read(relativeLogPath);
var query = new XQuery("for $logItem in doc(\"logs\")/record where $logItem/level==WARNING return $logitem/body");
var messages = logs.runQuery(query);

return messages;

or, better yet, if one exists, a LinqtoXML query.

notes:
- w3c intro to XQuery: w3schools.com/xsl/xquery_intro.asp
- Ranorex "User Code Actions" -- their mechanism for custom code integrated into tests: ranorex.com/support/user-guide-20/lesson-5-ranorex-recorder/user-code-actions.html
- getting XML formatted log outputs from `java.util.logging`: tutorials.jenkov.com/java-logging/configuration.html and oracle.com/javase/7/docs/api/java/util/logging/FileHandler.html
- this issue should very much be viewed as an exploration of .net! It is the other big programming collective after all! Use of fancy new non-java-friendly langauge features is expected.

Am i crazy to have Ranorex be log-sensitive?

Does this sound like the best approach? Is there a better strategy here than XML log files with XQuery searches? Does LinqToXML exist? is User Code Action the right facility in Ranorex? Has anybody got a public ranorex-using repo that does something similar?

I would've thought that there would be some knowledge of log files built into ranorex. I understand that Ranorex is supposed to be user centric, but even still, if our software is using a standard channel (logs) to explicitly express that it is in the error state, imho ranorex should not be stating that things are green-bar. But I cannot find a mention of a built-in-log-scanning feature anywhere. Can I make this a feature request?

Thanks for any help!

-Geoff

[Support Team]

Hi Groostav,

Unfortunately, there is no such a feature available out of the box. Ranorex is based on the .NET Framework. That means you can use all methods from this framework to achieve your intention.

In order to raise and discuss your feature request may I ask you to send it to [email protected]? We want to handle feature request via email in order to avoid misunderstandings.

Thank you in advance.

Kind Regards,
Matthias