Challenge
The company’s distributed team of test engineers must plan for testing in an environment where the number and size of releases in a given period depends not only on technical aspects, but also on commercial agreements with customers and the business model of the customers themselves.
Complicated test environment setup
Agile environment
With a goal of adopting a more agile process, the team is in the process of building out a Jenkins CI/CD pipeline. In addition, they have created an internal virtual environment to reproduce the customer environments, built on custom hardware using VMware as the virtualization environment.
High degree of customization
The company provides its customers a wide base of generally-applicable solutions, which it customizes to meet each customer’s unique requirements. These include adaptations for customers’ existing premises, environment, and production needs, and range from slight changes to the core behavior of software to more extensive modifications.
Result
The distributed team can now run tests for multiple configurations in parallel, as well as multiple software versions on the same configuration.
Prior to adopting Ranorex Studio, the main challenge for the team was to set up and reproduce a test environment that simulated the real production environment. At any given time, an customer might have multiple versions of their software installed across production sites, due to issues such as the time required to implement a new version or regulatory approvals. But testing on physical machines limited the possible configurations and the number of production scenarios that could be tested.
Now, the team can run multiple configurations in parallel, as well as multiple software versions on the same configuration. Either approach saves time and spreads test coverage among multiple possible platforms.
The team has also extended their test coverage significantly. “We are able to control and pilot all the applications simultaneously, because we instruct Ranorex Studio to do so”, explained the team’s manager, rather than having “4 or 5 people clicking manually in multiple locations.”
Executing tests in a virtual environment brings many advantages: the test engineers can connect the test environment remotely, without restrictions for the availability of hardware; and the system is qualified by a simulated production environment, using test data similar to real production data.
As a result, the team was able to achieve a 30% increase in testing productivity.
Recommendations
The immediate goal of the team is to fully adopt a CI/CD pipeline, thanks to the confidence that they have in their testing approach. The team primarily tests the UI through tests written in code rather than using recording modules. As the manager explained, “we used recording at the very beginning when we started to test the tool itself. We were not sure the extent to which we could control the application and the UI…. Recording is a good base when you have some problems in capturing objects or you don’t really know the exact syntax of some of the commands, etc., so passing through recording and converting to code can help. But on the other hand, we have developed a quite good set of libraries to abstract the object repository layer from the code. This allows us to maintain a test unaltered even when the XPath or the object itself changes. We are now in a situation where the same test is portable from a desktop application to a web application, or vice versa.”
