How to Reconstruct Failed Test Cases in CI Systems

When you want to run automated tests using different configurations, you will very likely use virtual machines. However, as with everything, you can run into problems. In this article, I’d like to demonstrate problems you may encounter when executing automated regression tests in CI environments on virtual machines that use different configurations and will show, how you can easily resolve these problems using Ranorex Remote.

• The starting situation (or rather, the problem)
• Solution 1: Reconstruct the error state manually
• Solution 2: Trigger Test Execution on CI server
• Solution 3: Execute Automation Steps Locally on VM
• Solution 4: Execute Automation Steps on the VM Using Ranorex Remote

Let me quickly set the scene:

The starting situation (or rather, the problem)

Let’s say you’re working in quality assurance in an agile environment. If so, you have likely encountered the problem of tests failing on specific virtual machines – and only on these machines. Locally – on your test-creation machine – the tests run smoothly without any error.

How can this happen?

Well, this could be due to several reasons, such as an unexpected pop up, like an update window, or a browser window has not been closed as expected on the virtual machine. Or maybe the system under test behaves differently than the local machine, or the test fails because the location-specific system settings, such the system language, differ from the one on the local machine.

So, how can you handle this error behavior?

Solution 1: Reconstruct state of error manually

The first approach that may come to your mind, is to manually reconstruct the preconditions leading to the error. This means that you have to go through the test report, locate the error and investigate the preceding steps, leading you to the point the error occurs. Once you’ve uncovered how the error can be reproduced, you can connect to the remote machine and repeat all steps necessary to exactly reconstruct the failure state.

This seems to be a legitimate approach if the complexity of the whole test workflow is limited. But the more steps you have to perform manually to reproduce the exact system state, the more time-consuming this procedure gets – and the less it makes sense to pursue this solution.

So, I think this approach is not practical in most use cases.

Solution 2: Trigger Test Execution on CI server

In the next approach, it seems only natural to replace the manual steps mentioned in the previous solution with automated steps.
The automated steps needed to achieve the actual state of your SUT when the error occurred already exist in your test automation project. To replace the manual tests, open the test execution build step on your CI system, and alter the command line arguments to only execute the failed test case. If you have a data connector bound to your test case, you should also update the data range, to reduce the test execution runtime to a minimum. For detailed information about command line arguments, please refer to the user guide section “Running Tests without Ranorex Studio”.

When following this approach, you can only run one test case. This means that you will not be able to run global setup and teardown sections or disable modules, etc., which may be necessary to restore the desired system state.

To overcome this restriction:

• add a new, or update an existing, run configuration, and update, if needed, the data range of your data connectors to reach the desired system state,
• commit your changes to the version control system,
• and trigger the test execution with the updated run configuration on the CI system.

This approach is problematic, as committing unnecessary changes to the version control system will mess up the revision history.

A non-transparent and complex version history is – especially when working together with a development team – not desirable.

Solution 3: Execute Automation Steps Locally on VM

Committing changes in the run configuration to the version control system is not required, because of the reasons mentioned above. So, it would make sense to run the automation locally on the virtual machine without using CI mechanisms at all.
Remotely connect to the VM and see if you can find the automation project on the hard drive. Then, adapt the run configuration and execute the updated automation workflow, so that only the failed test case will be executed.

But wait! In most cases, continuous integration systems clean up the executing machine after build and tests steps have been executed. This means that the probability of finding your test automation project on the virtual machine is rather low.Another possibility is to manually copy the test automation project to the virtual machine, which requires a shared network drive or similar.

There must be an easier and more comfortable way to reconstruct an error on the system under test!

What about using Ranorex Remote to solve your problem?

Solution 4: Execute Automation Steps on the VM Using Ranorex Remote

To do so, simply:

• install a Ranorex Agent on the system under test as explained in our user guide section “Install and Set Up Ranorex Agent”
• update your run configuration to only perform the steps necessary to reach the state the error occurs,
• and execute the test on the previously set up Ranorex Agent from the remote pad as explained in the user guide section “Ranorex Remote – Execute Tests”.

After performing these steps, the system under test should be in exactly the state the error has occurred. Now you can connect to the virtual machine and investigate the failed test case. In my opinion, Ranorex Remote is the fastest and most comfortable way to reproduce failed test cases on virtual machines.

Next to investigating failing test runs, Ranorex Remote is also very helpful during test creation. Simply execute your newly created test scripts on a remote agent. Continue using your machine in the meantime, as the test is executed on the remote machine and your computer will not be blocked. You’ll automatically be notified once your test has been executed and the report is ready. Re-check if everything works as expected, and commit your changes to the version control system.

Your revision history will be happy…