Test case sequence being reordered in MTM 2013 test run

Ask general questions here.
JimLin
Posts: 38
Joined: Mon Jun 02, 2014 4:23 pm

Test case sequence being reordered in MTM 2013 test run

Post by JimLin » Thu Feb 25, 2016 6:10 pm

Ranorex version 5.4.2
Windows server 2012r2 VM

I'm not entirely sure which sub-forum this question should go in.

We have an automated testing framework which, using CodedUI, links Ranorex (5.4.2), VSO 2013 (TFS) and VS 2013 on a dedicated automation server. It runs the tests in a sequence determined by a query in MTM which orders them in ascending work ID order and has been running fine until recently:
MTM test cases in correct order.png
MTM test cases in correct order.png (67.49 KiB) Viewed 1218 times
An MTM Test Run Log shows them being scheduled in the correct ascending test case ID order. In this image you can see the test case ID sequence going from test case ID 607 on line 9 to test case ID 624 on line 10:
test run ordering issue MTM log not broken before VS upgrade small.png
test run ordering issue MTM log not broken before VS upgrade small.png (58.75 KiB) Viewed 1218 times
In the last week we've noticed that although the order of the tests in the query hasn't changed, the tests are being reordered when the test run is being prepared in MTM:
test run ordering issue MTM log broken after VS upgrade small.png
test run ordering issue MTM log broken after VS upgrade small.png (59.87 KiB) Viewed 1218 times
In this second image, you can see the sequence change to test case ID 607 on line 9 to test case ID 25285 on line 10. The sequence continues to jump back and forth in a very disorganised way. This is causing us issues because some of the later tests rely on test data generated in earlier tests and therefore fail when this data isn't available because the test is running earlier than it should do.

Nothing has changed in the sequence of the tests in the query and we haven't added any new tests, but our devs have recently upgraded Visual Studio to Update 5 https://www.visualstudio.com/en-us/news ... e5-vs.aspx which contains some TFS updates and our problems started around the same time.

Has anyone come across this issue before or know whether it could be due to this recent VS update? I have hunted and hunted but have not been able to find anything similar on any other site. I have also posted this on Stackoverflow, but have had no response there.

Ultimately we can rollback to the previous VS but I am keen to understand what the issue is, as a rollback isn't necessarily as straight forward as it could.

jma
Posts: 111
Joined: Fri Jul 03, 2015 9:18 am

Re: Test case sequence being reordered in MTM 2013 test run

Post by jma » Fri Feb 26, 2016 3:20 pm

Hi JimLin,

I'm afraid that the issue doesn't seem to be related to Ranorex. Nevertheless, it would be great if you could update this thread if you find a solution to the current problem.

CreeTar
Certified Professional
Certified Professional
Posts: 13
Joined: Thu May 20, 2010 12:08 pm

Re: Test case sequence being reordered in MTM 2013 test run

Post by CreeTar » Wed Nov 23, 2016 3:47 pm

The order id in MTM is considered for manual test execution only, it is not used for automated execution of tests. The only way to use it would be to create your own test run using the console command tcm run /create and ordering the tests by sequence id. I already checked TFS 2015 Update 3, still the same. Also in VS 2017 is no new feature (such as TestNG dependsOn). The easiest workaround would be to use the
.orderedtest files introduced in 2010 and never changed again. They do not refactor, they
break upon renaming tests and they do not show individual results in Team Services or MTM, only
the .trx in VS shows the individual test results. Unfortunately Microsoft products are not used by
Microsoft, which leads to these "why wouldn't anybody fix this"-behavior throughout all Microsoft
products.