Test case sequence being reordered in MTM 2013 test run
Posted: Thu Feb 25, 2016 6:10 pm
Ranorex version 5.4.2
Windows server 2012r2 VM
I'm not entirely sure which sub-forum this question should go in.
We have an automated testing framework which, using CodedUI, links Ranorex (5.4.2), VSO 2013 (TFS) and VS 2013 on a dedicated automation server. It runs the tests in a sequence determined by a query in MTM which orders them in ascending work ID order and has been running fine until recently: An MTM Test Run Log shows them being scheduled in the correct ascending test case ID order. In this image you can see the test case ID sequence going from test case ID 607 on line 9 to test case ID 624 on line 10: In the last week we've noticed that although the order of the tests in the query hasn't changed, the tests are being reordered when the test run is being prepared in MTM: In this second image, you can see the sequence change to test case ID 607 on line 9 to test case ID 25285 on line 10. The sequence continues to jump back and forth in a very disorganised way. This is causing us issues because some of the later tests rely on test data generated in earlier tests and therefore fail when this data isn't available because the test is running earlier than it should do.
Nothing has changed in the sequence of the tests in the query and we haven't added any new tests, but our devs have recently upgraded Visual Studio to Update 5 https://www.visualstudio.com/en-us/news ... e5-vs.aspx which contains some TFS updates and our problems started around the same time.
Has anyone come across this issue before or know whether it could be due to this recent VS update? I have hunted and hunted but have not been able to find anything similar on any other site. I have also posted this on Stackoverflow, but have had no response there.
Ultimately we can rollback to the previous VS but I am keen to understand what the issue is, as a rollback isn't necessarily as straight forward as it could.
Windows server 2012r2 VM
I'm not entirely sure which sub-forum this question should go in.
We have an automated testing framework which, using CodedUI, links Ranorex (5.4.2), VSO 2013 (TFS) and VS 2013 on a dedicated automation server. It runs the tests in a sequence determined by a query in MTM which orders them in ascending work ID order and has been running fine until recently: An MTM Test Run Log shows them being scheduled in the correct ascending test case ID order. In this image you can see the test case ID sequence going from test case ID 607 on line 9 to test case ID 624 on line 10: In the last week we've noticed that although the order of the tests in the query hasn't changed, the tests are being reordered when the test run is being prepared in MTM: In this second image, you can see the sequence change to test case ID 607 on line 9 to test case ID 25285 on line 10. The sequence continues to jump back and forth in a very disorganised way. This is causing us issues because some of the later tests rely on test data generated in earlier tests and therefore fail when this data isn't available because the test is running earlier than it should do.
Nothing has changed in the sequence of the tests in the query and we haven't added any new tests, but our devs have recently upgraded Visual Studio to Update 5 https://www.visualstudio.com/en-us/news ... e5-vs.aspx which contains some TFS updates and our problems started around the same time.
Has anyone come across this issue before or know whether it could be due to this recent VS update? I have hunted and hunted but have not been able to find anything similar on any other site. I have also posted this on Stackoverflow, but have had no response there.
Ultimately we can rollback to the previous VS but I am keen to understand what the issue is, as a rollback isn't necessarily as straight forward as it could.