Integrating Bug Tracking with Your Test Automation Framework

You ran your test and noticed it failed. The first logical step is to open your issue tracking tool and create a bug ticket by manually entering the details of the failure. Wouldn’t it be much easier though, if you could create a ticket right from your test report every time a test fails? In this article, we will demonstrate how this can be done with a single click.

If we want to call our testing workflow efficient and successful, it is essential to have test reports generated once the test execution is complete. Learning how and why our tests failed is important to avoid future errors and improve the quality of our tests.

Creating a bug ticket for each failed test case is another important component of an effective testing workflow. It helps us to easily track and resolve all the issues and bugs we may encounter in our tests.

How is it usually done?

After test execution is finished, many testing tools generate some kind of test report to show us all important details regarding our test cases, i.e. if they failed, where they failed, the reasons why they failed, etc.
So, we take all those details and then manually create a bug ticket in our issue tracking tool.

While this is the common approach, it’s not the most convenient. Most of these reports are generated in the XML format, which makes things a bit more complicated. It can be hard to decipher the real reason why the test failed when you’re reviewing a chunk of code that isn’t always easy to read.

It would be much better if we could create bug tickets automatically from our test reports when the test case fails. This can be accomplished by integrating our test results with a third-party issue tracking or project management tools such as Atlassian Jira, Azure DevOps, or Bugzilla right in the testing environment. This gives us the option to create a bug ticket in case there is a failed test in the build.

Direct integration of a bug tracker into the testing environment means significantly less manual work since the reporting of detected problems in the testing environment is fully automated. This also implies that there would be an improvement in the productivity and quality of our testing process.

The question is, which issue tracking tool should we choose for this task? Well, it all depends on your preferences and needs, but the obvious choice for many is Atlassian Jira.
Atlassian Jira is the most popular issue tracking and management system among developers and companies worldwide. Jira’s capacity to manage issues, plan sprints, distribute tasks, customize workflow, track code, and production changes, as well as the ability to support test case management is what makes this tool essential for modern DevOps teams. Jira can also be easily integrated with other software and tools, including those used for test automation.

We’ll demonstrate how simple it is to connect your Jira project to a test automation tool and create bug tickets directly out of your test reports.

The test automation tool we’ll use in this example is Ranorex Studio, a complete test automation tool for desktop, web, and mobile. Ranorex Studio generates reports during each test execution. These reports allow you not only to view all imported details related to your executed test cases but also to automatically create issues for failed test cases in a Jira project and resolve issues after re-testing them.

To do this, we need to connect Ranorex Studio to our Jira project, which requires downloading and installing Ranorex Studio having a JIRA account.

To start, open Ranorex Studio and create a test project. Here is a tutorial on how to write your first test in Ranorex Studio.

Then, connect Ranorex Studio to Jira as follows:  first, click Tools > Jira integration > Start Jira wizard. The wizard opens with the Connect to Jira screen. For connection to Jira cloud, enter your Jira URL, Email, API token, and click Sign in.

The parameters in ‘Completed status name’ and ‘Active status name’ fields must match the values set for complete and active status in your Jira project in order to allow you to also reopen issues right from your test reports.
When you click the ‘Connect’ button, you’ll be prompted to enter your Jira username and password.

If you use Jira Server edition, enter your Jira username in the ‘Username’ field. If you use Jira Cloud edition, enter your dedicated email address for the username as explained in this Atlassian Community post.

Required parameters for the ‘Password’ field also vary depending on whether you are using Jira Server or Cloud edition. The Server edition requires your Jira account password, while Jira Cloud edition requires an API Token to be created.

To create an API Token, do the following:

1. Log in to https://id.atlassian.com/manage/api-tokens
2. Click ‘Create an API token.’
3. Enter a ‘Label’ for your token that you can easily remember and click ‘Create.’
4. Click ‘Copy to clipboard,’ then paste the token to the “password” field in the JIRA integration dialog.

Now that we have connected Ranorex Studio to Jira, we can add an endpoint to run our test. An endpoint is a set of configurations about the runtime environment of your test, such as which browser or operating system to use, and much more. In Ranorex Studio you can define local, remote, or custom endpoints to run your tests.

Save the endpoint and run the test. Upon completion of the test execution, a test report will be generated. If the test case has failed, we can create a Jira issue with a single click.
To do so, we’ll open the report and navigate to the failed test case, expand it and click the ‘Create Issue’ button.

This allows us to open the Jira issue in our default browser.

Ranorex Studio also allows us to resolve the issue directly from the test report, for example, when the same test case was successful after a repeated test run. In this case, simply click the ‘Resolve Issue’ button to close the Jira issue.

Of course, it’s not uncommon for bugs to show up again. For this reason, we can reopen the issue by clicking the ‘Reopen Issue’ button in the test report. However, your Jira project must support this workflow transition. Again, make sure that both ‘Active’ and ‘Completed’ status in the Preference dialog match the values set in your Jira project to allow reopening issues.

It should be noted that test cases in Ranorex Studio can only connect to one Jira issue at a time. The link information (additional data) is stored next to the test case in the form of a code comment, which includes all necessary information and can be edited if necessary. The link information will remain in the test case code even after the issue has been resolved.

Conclusion

In this example, we demonstrated how easily you can simplify and speed up your test workflow with the ability to generate JIRA issue tickets right from your test reports. This approach also enables you to resolve the issue from the test report as well as reopen the issue if the bug appears again.

Having test reports generated in a visually comprehensible format, as well as the integration with a JIRA project to create issue tickets right from those reports are just a few of the many benefits you get if you choose Ranorex Studio as your test automation tool.