Using Ranorex With VS & TFS 2010

Ask general questions here.
DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Mon Mar 28, 2011 2:39 pm

Hi,

I've been tasked with doing some research into the integration possibilities of Ranorex into the Visual Studio 2010 and Team Foundation Server 2010 (including Lab Management 2010) suite.

Are there any links you can provide where I can read up more on this topic? If not, please provide information on the following (I will add more questions once these has been answered and/or when I think of more):

1) When using VS's automation tools, integration with Lab Management & TFS is provided by default (this goes wihtout saying really). More specifically, I know that Ranorex tests compiles into EXE's - and that can obviously be integrated into any build process quite easily. But I cannot find any information on how results of the tests can be integrated into results of the Build process in VS? For example, when using the Lab Management build template in VS, the process automatically sends of one or more e-mail after a test-run containing detailed information on the test-run and its result(s). How does the VS build process get these results when running a Ranorex test (in EXE format or any other format)?

2) Licensing - what licenses would we need to record tests on one machine only, and integrate these tests onto a TFS server and also deploy them via Lab Management to multiple VMs? Is the license for recording and running tests seperate?

Hope to hear from you soon.

Regards,

Danie Spreeth
Senior C# Developer
Kaplan Learning Technologies

User avatar
Support Team
Site Admin
Site Admin
Posts: 11710
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Using Ranorex With VS & TFS 2010

Post by Support Team » Mon Mar 28, 2011 10:33 pm

Hello Danie,

There is a new blog addressing this: Combining Ranorex 3.0 and Visual Studio 2010 for UI Unit Testing

In the blog the test suite is transformed into test method's of a VS test project. The reporting thus is that of Visual Studio. The test result details do contain all Ranorex Logging as well.

To run the resulting VS project you need the Ranorex Runtime License, i.e. technically the Ranorex DLLs.

Note: The Ranorex Project format is that of VS2008. It can also be opened in VS 2010. This way you can also commit it to TFS. You can also use MSBuild to compile it. That a VS test project referencing Ranorex Assemblies can be committed to TFS is obvious anyway.

Regards,
Roland
Ranorex Support Team
.
Image

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Tue Apr 05, 2011 3:46 pm

Hi Roland,

Thanks for the response.

It took me a while to get everything running the way your blog describes, but I eventually got there. I now have a script that automatically creates VS TestMethods for each test inside of the provided rxtst file. When deploying to a VM, they all run as they should.

However, I still do not see the Rabnorex reports integrated into the VS reports (or rather Test Manager Reports since TM is where you view the results) after the build process.

Is there any step somewhere I might have missed not metnioned in the blog you provided on this subject?

Below is a copy of one of the TestMethods being executed. I can physically see the tests running on the VM, so they definately execute, its just the Ranorex reports that are missing.

Code: Select all

		[TestMethod]   
		public void Test_SlideNotes()   
		{   
		    int res = Ranorex.Core.Testing.TestSuiteRunner.Run(typeof(FlexEngineAT.FlexEngineATRepository),
                                   "/testcase:SlideNotes  /param:\"StartPath=" + SutPath + "\"", m_RanorexTestSuiteXML);   
		    Assert.AreEqual(res,0); //-1 is return on failure   
		} 
Just a quick note on the code above: The member 'm_RanorexTestSuiteXML' is being populated before the tests are called with the rxtst xml and the sutpath is also provided.

Thanks

Danie.

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Mon Apr 11, 2011 9:51 am

Hello??

It's been almost a week now with no response... The whole exercise around executing Ranorex tests through VS Test Methods was to be able to see the Ranorex reports in TFS/VS2010.

Please repond?

Thanks

Danie Spreeth

User avatar
Support Team
Site Admin
Site Admin
Posts: 11710
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Using Ranorex With VS & TFS 2010

Post by Support Team » Mon Apr 11, 2011 10:39 am

Hello Danie,

we seem to have missed your post. I'm very sorry.

The Ranorex Report is generated but not integrated in the Visual Test Project result.
Only the Ranorex logging can be seen in the Visual Test Result details.

From the bin folder in the console I did:

Code: Select all

mstest /testcontainer:VIP_VS_Test.dll /test:Test_AddAndDeleteSingleVIP
Then, below the bin there is a TestResults folder with trx files.
When you open the .trx file, right-click and choose View Test Result Details from the context menu, all the console output done by Ranorex is there.

The trx file has a folder associated to it, below the TestResults folder, which contains the Out folder and there is the ranorex report.

Regards,
Roland
Ranorex Support Team
.
Image

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Mon Apr 11, 2011 10:43 am

Hi,

Thanks for the reply.

The trx files you are referring to and the TestResults folder as well does not exist on my side. The only difference between what you are doing and what I am doing is that you are manually executing the tests using MSTest.exe. Here, I have a build that executes the Tests automatically (which I am sure also uses MSTest in the background).

I have all my test methods created as per the example in my previous post and these test methods are linked to actual TFS Test Cases. The build then executes all automations linked to a predefined Test Plan in MTM.

Where do I now find these reports?

User avatar
Support Team
Site Admin
Site Admin
Posts: 11710
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Using Ranorex With VS & TFS 2010

Post by Support Team » Mon Apr 11, 2011 12:30 pm

Hello Danie,

if you start the tests from within Visual Studio the TestResults will be below the project folder.

Else please search your disk for trx or rxlog files. Then you should find the location for your case.

Regards,
Roland
Ranorex Support Team
.
Image

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Wed May 04, 2011 8:05 am

Hello again Roland,

Regarding my previous post, I found the rxlog files under C:\Users\CurrentUser\... - so that is sorted. I now have a different issue though.

One of our other developers have been in contact with you a few weeks ago, and they changed the structure of the Ranorex projects as per discussions with you. Previously, compiling the Ranorex project produced one EXE, now they have each test in it's own DLL. I managed to change the C# code in the VS Test Project so that the tests still starts, but now I am getting errors with ALL tests.

NOTE: Previously when there were no seperate DLLs, all tests ran fine...

When running I get the error saying that:

"The module 'xxxx' (with id {guidgoeshere}) could not be found.
Stack trace: at Ranorex.Core.Testing.TestSuiteModule.RunInternal(DataContext parentDataContext)"

My C# code calling the tests is as follows:

Code: Select all


[TestMethod] // For Test Case ID 1877   
public void Test_SlideNotes_1877()   
{   
    int res = Ranorex.Core.Testing.TestSuiteRunner.Run(typeof
                                                                      (SlideNotes_1877.SlideNotes_1877Repository),   
		                                      "/testcase:SlideNotes_1877  /param:\"StartPath=" +     
                                                                       C_SutPath + "\"", m_RanorexTestSuiteXML);   
    Assert.AreEqual(res,0); //-1 is return on failure   
} 

I have ensured that all DLLs and the Ranorex compiled EXE are added as references to the project as well and also gets copied to the build output path.

Please help URGENTLY! We have a major demo tomorrow morning demonstrating the Automated Tests to management.

Thank,

Danie.

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Wed May 04, 2011 2:53 pm

Hi,

Still hoping for a reply soon on this...

Meanwhile I manually copied all test files and dlls to a location on the VM I am testing on, and manually try to execute using mstest.exe hoping that something was wrong with my build process. Getting the same result (one good thing is that my build process them seems to be ok).

This is a part of the .trx file produced by mstest.exe - maybe it will help with solving this:

Code: Select all

<UnitTestResult executionId="2f6239ff-42ce-41ae-8335-a216d0f5304e" testId="93ffd3d6-72c7-e81c-0d6f-245071564a1b" testName="Test_CourseCompletion_2054" computerName="AL3TESTS32" duration="00:00:01.5174172" startTime="2011-05-04T15:45:45.1265637+02:00" endTime="2011-05-04T15:45:46.9742190+02:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Failed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="2f6239ff-42ce-41ae-8335-a216d0f5304e">
      <Output>
        <StdOut>[2011/05/04 15:45:46.177][Debug  ][Logger]: Console logger starting.
[2011/05/04 15:45:46.302][Info   ][Test]: Test Case 'CourseCompletion_2054' started.
[2011/05/04 15:45:46.390][Info   ][Test]: Test Module 'Launch' started.
[2011/05/04 15:45:46.475][Error  ][Module]: The module 'Launch' (with ID {3738d4af-8903-473b-b506-1e73d1a0a166}) could not be found.
[2011/05/04 15:45:46.476][Failure][Test]: Test Module 'Launch' completed with status 'Failed'.
[2011/05/04 15:45:46.496][Info   ][Test]: Test Module 'Quit_Browser' started.
[2011/05/04 15:45:46.540][Error  ][Module]: The module 'Quit_Browser' (with ID {0f373906-1c17-4ec5-b44a-a43fbec3c59b}) could not be found.
[2011/05/04 15:45:46.540][Failure][Test]: Test Module 'Quit_Browser' completed with status 'Failed'.
[2011/05/04 15:45:46.549][Info   ][Test]: Test Module 'Restoring_Default_Settings' started.
[2011/05/04 15:45:46.580][Error  ][Module]: The module 'Restoring_Default_Settings' (with ID {260651db-f036-44d7-b765-fe455540e197}) could not be found.
[2011/05/04 15:45:46.581][Failure][Test]: Test Module 'Restoring_Default_Settings' completed with status 'Failed'.
[2011/05/04 15:45:46.581][Failure][Test]: Test Case 'CourseCompletion_2054' completed with status 'Failed'.</StdOut>
        <DebugTrace>W, 43192, 14, 2011/05/04, 15:45:46.941, 5445906446938, QTAgent32.exe, IEDOM : StopSession of the plugin called before StartSession
W, 43192, 14, 2011/05/04, 15:45:46.942, 5445906453250, QTAgent32.exe, UIA : StopSession of the plugin called before StartSession
</DebugTrace>
        <ErrorInfo>
          <Message>Assert.AreEqual failed. Expected:<-1>. Actual:<0>. </Message>
          <StackTrace>   at FlexEngineTests.FlexEngineTests.Test_CourseCompletion_2054() in c:\sourcetemp\testproject\rx_FlexEngineTests.cs:line 33
</StackTrace>
        </ErrorInfo>
      </Output>
      <ResultFiles>
        <ResultFile path="AL3TESTS32\Test_CourseCompletion_2054.png" />
      </ResultFiles>
      <ExtensionResult>Exception has been thrown by the target of an invocation.</ExtensionResult>
    </UnitTestResult>
Where should these modules be located then? They are currently in the folder I executed mstest from and I also recreated my local machine folder structure (to have the modules in same place the original test project references them from - just to try and find the problem - but no go.

Please Ranorex - I am running out of time and ideas here...

Thanks

Danie Spreeth.

User avatar
Support Team
Site Admin
Site Admin
Posts: 11710
Joined: Fri Jul 07, 2006 4:30 pm
Location: Graz, Austria

Re: Using Ranorex With VS & TFS 2010

Post by Support Team » Wed May 04, 2011 7:29 pm

Hello Danie,

I suppose they have made DLLs and then used recordings and user modules from those DLLs in the main executable's rxtst file.

I have tried it with a very simple configuration DLL + EXE with executable's rxtst using a recording from the dll and a recording from the exe. It worked straight away from within VS and using MSTest. The Ranorex executable and dll are referenced in the VS 2010 project. The test case was started like this:
int res = Ranorex.Core.Testing.TestSuiteRunner.Run(typeof(testdllexe.testdllexeRepository),"/testcase:TC",Resources.testsuitefileresource);
Assert.AreEqual(res, 0);//-1 is return on failure
I once had a similar problem and it was associated with the typeof(...) parameter. With the above repository from the executable it worked in my little dll+exe sample. Please do some trial-and-error with that parameter.

Other than that, what is the Ranorex version you are working with? I use 3.0.1.

Regards,
Roland
Ranorex Support Team
.
Image

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Thu May 05, 2011 7:31 am

Hi,

My code is exactly the same - there is just one thing I am not sure of.

The "repository" you are calling, does that live in the DLL or the EXE? In my case there are no repository in the EXE, only in the DLLs. From your code I cannot figure this out since you have "dllexe" in the name... "testdllexe.testdllexeRepository"

My demo is in less than 2 hours...

Danie.

PS: If I execute the tests directly from within visual studio they also dont work - nor do they work locally on my dev machine when i execute them using mstest directly. When I run them using the compiled EXE they work fine though...

PPS: I am also using version 3.0.1
Last edited by DanieSpreeth on Thu May 05, 2011 8:51 am, edited 1 time in total.

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Thu May 05, 2011 8:19 am

Hi,

Let me just make sure you understand exactly how these tests were created in Ranorex Studio...

We have 6 test cases in one suite. All of these 6 tests uses (for startup and shutdown) the same common repository which is in its own Ranorex DLL. So we have

1 x EXE
1 x common.dll
6 x testcases.dll

I have now also tried referencing the common repository in the typeof param when calling all 6 tests, now the parts of the tests that is inside the common repository (common steps) executes, but the main parts dont.

I am just wondering if the structure we are using is not maybe the problem. Since each test actually uses 2 DLLs, yet in thr Run command we pass in typeof just one of them?

Danie.

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Thu May 05, 2011 10:07 am

Hi again...

I am now pretty sure I know exactly where the issue is.

As per my last post above, when i pass in the Common repository into the TestSuiteRunner.Run method, only the common steps executes, but when passing in the actual test's repository, it fails because it cannot find the steps in the common repository.

As I see it 1 of 2 things must happen here.

1) You must give me an alternative method of calling the Ranorex tests as we have structured it so that we can pass in both repositories (or something of the sorts).

2) This one is a no-go for us since it is against all OOP principles. We must copy all common steps into each test dll. This will lead to duplicate code which is not acceptable for obvious reasons.

Please assist.

Danie.

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Thu May 05, 2011 12:20 pm

Hi

I figured out an alternative method to execute the tests from VS, an even though it works for us for now, it is not optimal.

Code: Select all


try
            {
                AL3RuntimeEngineTestSuite.Common.StartUp.Launch.Start();

                SlideNotes_1877.Editing_Notes.Slide_Notes_XML_Editing.Start();
                SlideNotes_1877.Editing_Notes.Validating_Slide_Notes.Start();

                AL3RuntimeEngineTestSuite.Common.Quit.Quit_Browser.Start();
                AL3RuntimeEngineTestSuite.Common.Restore_Default_Settings.Restoring_Default_Settings.Start();
            }
            catch (Ranorex.ValidationException)
            {
                Assert.Fail();
            }

Pls let me know if there is better way, and/or fix the TestRunner API to be able to run tests in multiple repositories.

Hope to hear from you soon.

Danie.

DanieSpreeth
Posts: 13
Joined: Mon Mar 28, 2011 2:29 pm

Re: Using Ranorex With VS & TFS 2010

Post by DanieSpreeth » Thu May 05, 2011 3:39 pm

What a day! :shock:

I rewrote my Test Project Generator to call all tests as per my previous post. In order to get that right, I read the correct order, module names and namespaces from the rxtst file, and then using Reflection, find the correct method call and generated the code with that.

Kind of chuffed with being able to get it working so quick :D

Still, I would like to know if this way of calling the tests is the best available way or not ?

Regards,

Danie.