How to Transition from Manual to Automated Testing: Idera DevOps Tools Podcast Episode 2.1

Software testing is essential for identifying and fixing defects before going to market. The more efficient the testing, the better. That’s why you might be considering a move from manual to automated testing.

Manual testing requires a human tester to physically perform a series of steps to detect errors in functions of an application.

Automated testing is the use of software to test software. Automated testing uses tools to test the software functions and compare the results to predicted results. This can provide your team more time to concentrate on other important aspects of their work.

Automated testing streamlines the testing process, saving time and money as well as providing a better quality product. In a 2016 survey, 24% of respondents saw a return on their test automation investment within the first 6 months.

But moving from manual to automated testing can seem daunting. In this article, you’ll find a transcript of our recent podcast on how to make the transition.

Transitioning From Manual to Automated Testing: Podcast Transcript

Jackie King: Hi everyone! Welcome to the latest episode of the Idera DevOps Tools podcast. Our goal is to educate and inform you about key topics in software development. With solutions that help almost one million users throughout every step of building, testing, and deploying applications, our experts are poised to provide enticing insights, perspectives, and information. I’m Jackie King with Ranorex. Joining me for today’s roundtable are TestRail Product Manager Simon Knight, Ranorex product manager Jon Reynolds, and José Domingues from XRay.

Our topic today is “transitioning from manual testing to testing that’s supported by automation.” With all the challenges that companies have faced over the past two years, and continue to face going into 2022 — from lockdowns and remote work to staff turnover and shortages — automation-supported testing has a lot of appeal.

Simon Knight: Hi everyone. As Jackie said, I’m Simon Knight, Product Manager at TestRail. I’d like to take a moment to talk about what we mean by “automation-supported testing.” Sometimes in this industry you hear terms like “automated testing” used in a way that make it sound like automation is replacing humans in testing. But we don’t think that’s really what test automation is about — it’s about automating all of the repetitive stuff so that humans have got more time to concentrate on the things that they’re good at: the creative work — the thinking and exploring of an application to really understand what it does.

So one example of that might be cross-browser testing. Another one might be regression testing, and another good example is testing across lots of fields into which you want to put a variety of data values.

Jon, my colleague on the Ranorex side probably has a bunch of thoughts on that as well, especially since Ranorex is a test automation tool.

Jon Reynolds: Thanks Simon, So what we’ve seen is that teams get the greatest return on test automation investment by focusing on the right test cases — like you said, automating repetitive work, performing the same tests across multiple browsers, regression tests, or data-driven testing. So repeating tests just using different datasets over and over again. And that ROI, or return on investment, can be measured not just by productivity improvements, but also in measures of quality and user satisfaction. So as these testers have more time to focus on other things, they can more thoroughly test the product by automating the tasks that they don’t really need to do manually.

But since ROI is really the end of the story when talking about transitioning from manual testing to automation-supported testing, let’s take a step back, and take a look at what it takes to have a successful automation project.

So the first step is to complete a readiness assessment, to ensure that your team is ready for an automation project.

José Domingues: That’s right Jon. So, I’ll jump in on that. You really need to understand why you need test automation — what problems you’re trying to solve by automating. So, for example, has QA become a bottleneck in your build and release pipeline? Or, are you seeing too many errors in production? Once you identify the problem that you’re trying to solve, then you’ll want to set a definition of success that is specific and measurable. I think this is really important in terms of basic project management.

It’s important to realize that automation won’t fix a broken process. If QA is a bottleneck because there isn’t enough communication with members of the development team, automation won’t fix that. So your readiness assessment should include a review of the existing processes, as well as ensure that you’ve got resources in place, such as test devices, data, and environments. But most importantly, you’re going to need one or more champions on your team who are excited about automation and willing to drive the project forward.

And lastly, think about how you’re managing your testing now. In addition to a test automation tool, do you need a test case management tool like TestRail or XRay, for example?

Simon Knight: Of course you do, José! But after a readiness assessment, the next step is to put together the right team. As José mentioned, it’s important to have one or more test automation champions on the team. Other essential team members could include representatives of stakeholder groups, developers, testers, business analysts, and sysadmins, those kinds of people. And this team should ideally meet on a regular basis to set their goals, monitor progress of the automation efforts once they get underway, and adjust strategy if and when necessary, and ultimately to approve the final solution.

We also recommend preparing a handful of cases for test automation. Good candidates for an initial set of automated test cases would most definitely smoke tests, build acceptance tests, and probably your high-risk regression tests as well. Ideally, we’d recommend including at least one end-to-end test case, to ensure the selected test automation tool works with your key functionality — a steel thread test case, as I’ve heard Lisa Crispin refer to it in the past.

Jon Reynolds: Thanks Simon. So, really the next step is to research and select two or three tools to conduct a proof of concept. A proof-of-concept is just a short-term project. Typically, you take a few weeks where you evaluate the leading tools in your environment, and then measure them against your evaluation criteria, to determine whether or not that tool will meet your needs as you transition to more automation-supported testing.

At Ranorex, we have sales engineers that can assist you during your hands-on evaluation. You can also begin to calculate expected return on investment through an automation tool.

José Domingues: Once you have selected an automation tool, the process of implementing your automated tests can begin right away.

Set up the test environment and configure any necessary integrations, such as defect tracking applications like Jira, test management solutions we’ve mentioned before like TestRail or XRay, CI/CD servers, and so forth. Because test automation is a software development project on its own, consider using a source control system such as Git for your test cases.

Be sure to allow time for your team to get some training and accommodate the learning curve of shifting away from manual testing. Up-front you should expect to see more effort and investment in learning, which later will lead to productivity improvements. To speed up the learning curve, investing in professional training services is also an excellent option, and there are tons of partners around the world that provide these services. Our recommendation would be to find someone close to you, as that usually works best.

Simon Knight: Thank you, José. We also think you should treat your automated test cases with the same love and care that you would your application code and also your manual test cases. To the extent that it’s necessary to do so, bearing in mind that sometimes the code itself might be the documentation, you should add additional supporting documentation where it’s necessary to do so. And this is where tools like XRay and TestRail can help out.

Definitely consider using a version control system like Git to help protect your automation code from accidental changes or conflicts when you have lots of different people working on it, and even from losing it as well. I’ve seen that happen in the past: where people have lost their code completely — actually including myself, now that I come to think of it — because I didn’t check it into a source version control tool. So that’s definitely something that you’re going to want to do.

Jon Reynolds: I agree. There’s not much worse than losing all the work that you’ve done due to something you did.

So after working on and starting with your proof of concept, and then finishing your proof of concept, the final step is to build on the success of this project. So once you’ve selected a tool, start using an iterative process to maintain and expand your automation suite. In each iteration, you review what worked and then what didn’t work, adjust your strategy as necessary and repeat. You can build on areas where automation is really starting to help. And then you can also begin to calculate the actual ROI that you’re getting from this tool you selected.

Jackie King: So now that we’ve talked about the key stages of a test automation project, let’s take a quick look at some barriers that teams encounter when transitioning from manual to test automation. In partnership with Pulse research, we recently polled 94 IT leaders, and asked them to identify the top three barriers to automating their user interface testing (shown below).

Barrier #1: Cost of test automation tools

The number one barrier, according to this survey, is the cost of automation tools. And I think that’s really interesting, because of the popularity of free and open-source tools — I didn’t assume that cost would be the top barrier. Of course, there are hidden costs in open-source tools that can really raise the total cost of ownership, so to speak — sort of like getting a free puppy.

José, do you have any thoughts on ways to overcome the barrier of cost of test automation?

José Domingues: So yes, Jackie. You mentioned that there might be hidden costs when looking at these tools. It’s not just how much it costs per user for the tool that you’re using.

So the way to overcome that barrier is to do an ROI analysis before jumping in. Consider all of the potential costs of the automation vs. the costs that you currently have with manual testing. And here we’re not just talking about costs such as hardware, software, training, and payroll – but most importantly, what you incur from releasing code with defects. A major bug that goes out in the world to your users might have a huge impact, a lot bigger than the fees that you pay for an automation tool.

Jon Reynolds: Yeah, I agree. So one thing that we have at Ranorex is a free ROI calculator template that you can download from our website. It considers variables such as the size and expertise of your team, the number of tests that are performed manually, and the cost to perform those tests manually vs. the cost to automate them. Once you plug in the numbers, the template calculates the total time and labor costs for the first execution of your manual test, and then for subsequent cycles.

To give you an idea of the ROI that you can expect, over 80% of Ranorex customers report that they’ve seen a 20 to 40% increase in productivity after adopting test automation.

Barrier #2: Lack of automation expertise on the team

Jackie King: So in the survey on the barriers to UI test automation, the number two result was lack of automation expertise on the team.

Simon Knight: I think that’s kind of a common problem. This goes back a bit to our earlier suggestion about maybe having a test automation champion on the team. That person could be a full-time member of your staff or they might be a consultant or a contractor of some form. It’s important to say that the champion doesn’t necessarily need to be a kind of rock-star test automation all-singing, all-dancing architect type, especially if you are using a relatively low-code/no-code approach — using a tool like Ranorex, for example — but probably what they do need to be is someone who is sufficiently passionate and enthusiastic about the project to be able to drive it forwards and through to completion, and they should be sufficiently skilled to do the work from a technical standpoint.

Jon Reynolds: And what we’ve found is a tool that has low-code and no-code functionality can actually be a springboard for helping your team gain those automation skills. So as you can tweak those — you initially set up your tests using a low-code or no-code setup. If the tool also provides the option to manipulate those tests using code, your testers will start learning how to do some of those coded automation tests as well.

Barrier #3: The number of tests to be automated

Jackie King: The third barrier identified in the survey is an excessive number of tests to be automated.

José Domingues: My recommendation here would be to start small and iterate on that. Usually what we find is the ambition to turn everything into automated tests, go immediately to 100%, and that’s not easy and maybe not always possible, or shouldn’t even be the ambition that some teams should have. Instead, focus on the most-repeated tests or tests for critical features. You can even do a small risk analysis to prioritize what those critical cases are, and try to automate those as a starting point.

Another common approach would be to automate your smoke tests or focus your longer, scenario-based tests, as these have a lot of value for finding defects.

Barrier #4: Not enough time for test automation

Jackie King: So Simon, the fourth barrier that the survey reported is not enough time in the development cycle for UI test automation

Simon Knight: Not enough time, Jackie? Surely not! Who would have thunk it?

But in all seriousness, time is always going to be a challenge, you know. It is a difficult one. A common concern of testing teams (even our own testing team) is that there isn’t enough time to do all the testing that they want and that needs to be done, whether that testing is manual or automated. I think this is true in teams that are practicing agile or some form of lean development approach, and it’s equally true for waterfall approaches. The truth is that testing is never done. As the old saying goes, “you can prove the presence of bugs, but you can’t prove their absence,” certainly not by further testing.

Specifically with automation, the challenge is that you can’t start creating UI-focused test automation until there’s a UI to actually test.

While a lot of teams talk up DevOps and continuous deployment, what we see in practice is that a lot of teams run sprints that are really a series of mini-waterfall cycles. If your team is working in two-week sprints, for example, you could approaching this problem by automating the tests that you plan to repeat from the last development cycle at the same time the rest of your team is working on new features. And user interface tests for new features released in the current cycle can then always be done manually, at least at first.

Barrier #5: Flakiness or unreliability of existing automation

Jackie King: Jon, I’m going to toss the fifth barrier over to you, and that is the “flakiness or unreliability of existing automation.”

Jon Reynolds: So flaky or unreliable tests can set teams off on the wrong foot when they’re trying to transition to automation. When you’re starting with an automation project, they reduce your team’s confidence in the tests. Your testers may just start disregarding the results, or stop running your automation altogether.

So there are some things that you can do when you are writing tests to make them more stable from the start. The first is to use a tool (say Ranorex) that has built-in intelligence to keep working even when you have minor changes in your UI. But regardless of the tool you’re using, your tests will be more reliable and easier to debug if they are small and modular.

So try to eliminate dependencies as much as possible – meaning that one test shouldn’t depend on another test completing successfully. Be sure that each test contains the necessary setup steps to prepare the system before the test runs, such as creating test data. Each test should also perform a “teardown” to clean up the system after it runs.

And be sure to test your tests! Run them multiple times before adding them to your regression suite or whatever functional area that you’re testing. Run them in different orders as well, this will help you be sure you don’t have hidden dependencies on your application.

Jackie King: Thank you Simon, Jon, and Jose, for discussing how to transition from manual to test automation with us today, and thank you to everyone who’s listened in. Be sure to subscribe to our podcast so that you don’t miss any of our upcoming episodes.

If your team is looking to save time and money as well as create repeatable tests that improve product quality, automated testing is the answer.

Ranorex is a leading provider of specialized and comprehensive test automation tools that help you overcome your testing challenges. To learn more about why Ranorex is the solution for your organization’s test automation needs, contact a test automation expert today.