The profession of software testing is undergoing rapid change, driven by a transformation in software development practice. Until recently, the “waterfall” model of software development has been the leading practice. According to Gartner, Inc., “waterfall methods were employed on 56% of development efforts in 2015, with iterative methods used in 21% and agile in 23%.”
Now, agile teams are in the majority. In a recent online survey, just over 80% of professional developers reported using agile methods, compared to 27% using waterfall methods. The proportion of agile development was even higher in an email survey of over 230 testing professionals conducted by Ranorex in August 2017, where 90% of respondents reported following agile practices. However, the transition to agile is incomplete: 8% of respondents to the Ranorex survey describe their development practice as still “primarily waterfall,” while another 35% are using a combination of agile and waterfall practices.
The goal of an agile team is to increase customer satisfaction through rapid delivery of a high-quality solution. Certain practices are common to all forms of agile:
Within these broad principles, there is wide variation in how teams practice agile development.
Respondents could choose any number of primary and secondary practices, or none at all. They reported employing an average of 1.8 primary practices and 2 secondary practices. Kanban is the most common secondary practice at 26%, followed closely by Test-Driven Development (TDD)/Acceptance Test-Driven Development (ATDD) at 25%. Industry-wide, Kanban is used in combination with Scrum so often that the practice has its own name: ScrumBan.
“We…work in small focused teams following bits of Scrum, bits of Kanban, with a healthy dose of Extreme Programming…. Our cross functional teams have Product Management, all aspects of Product Development (including DevOps and Test), and UX all working together on a daily basis… The team works very closely, and like to pair program on pretty much everything.”
In the Ranorex survey, 11% of respondents reported using DevOps as a primary development practice, and another 18% as a secondary practice. This is similar to the results in the 2017 Stack Overflow survey, where 11% of developers identified themselves as “DevOps specialists;” the same survey reported that DevOps specialists are the highest paid developers worldwide, well ahead of mobile and web developers.
A key indicator of interest in a subject is the number of related web searches. A Google Trends analysis shows that searches for the term ”DevOps” have increased steadily since 2012, reaching their current peak in the summer of 2017. By comparison, interest in the term “Scrum” during the same period remained relatively unchanged.
DevOps takes the principles of Agile and expands their scope, recognizing that ensuring high-quality development requires continual engagement and feedback from a variety of technical experts, including QA and operations specialists.
The shortest iterations are found in CI/CD and DevOps environments, where new code may become available for testing every day – or even several times per day. Testing in continuous environments presents unique challenges. First, tests must be ready to run as soon development completes, which means that testers might begin writing their tests while developers are still working on the code. Tests may have to be re-written quickly in response to changing requirements. In addition, automated tests must complete in a very short time frame, so it may be necessary to execute them on multiple remote systems, in parallel.
Given these challenges, continuous testing environments benefit from a combination of automated and manual tests, as shown below.
In the Ranorex survey, respondents on agile teams reported that they commonly create test artifacts such as test cases, participate in the creation of test plans, and help identify areas to test. However, fewer respondents said that they were able to help write testable user requirements or define acceptance criteria. On fully agile teams, these numbers should be closer to 100%.
During development, testers and developers on agile teams should work in pairs to ensure that effective unit testing is occurring; and that testers understand the product and its requirements to support exploratory testing. However, only 22% of respondents to the Ranorex survey reported that they assist developers in this way. Just 11% of respondents regularly coach developers in testing techniques. More commonly, testers create automated test cases (54%), conduct exploratory testing (51%) or scripted testing (46%), and analyze completed tests (53%).
Our experience with agile teams has shown that testing skills and experience are vital to project success, and that testers do add value to agile teams.
35% of respondents to the Ranorex survey are in “mixed agile and waterfall environments.” Testing can be negatively affected when a sprint becomes a mini-waterfall, with activities occurring in shortened but still structured phases. Testers may have nothing to do at the start of the sprint, and then be rushed at the end of the sprint. Or, testing may be delayed until the start of the next sprint. To avoid this issue, system and acceptance testing should occur as each story is completed, rather than for all stories at the end of a sprint – or worse, in a special “testing” sprint, as shown below:
Many waterfall practices don’t work well in combination with an agile environment. Foremost among these is keeping QA and development as separate groups. For an agile environment to be successful, testers should function as full members of the development team. QA should contribute to sprint planning so that there are adequate resources for testing, participate in customer/business meetings, help develop testable story requirements, and partner closely with developers throughout the iteration window .
In a “three amigos” meeting, a product owner, developer, and tester work together to create testable user stories and acceptance criteria.
The increasing emphasis on unit and integration testing sometimes raises questions about the relative importance of traditional GUI testing. However, the ideal mix of testing for a given project depends on the nature of the application and the level of risk. A mobile app may need a higher volume of automated GUI tests relative to unit and API tests since much of its functionality is in the GUI. In addition, unit and integration testing don’t address important aspects of user experience (UX) and usability testing.
While 70% of positions asked for automated testing experience, 41% specifically mentioned “manual testing” as one of the testing skills either required or preferred. 74% of positions listed structured testing experience, such as experience in writing and executing test cases or test scripts. These results demonstrate that even as the separation of roles between developer and tester is decreasing, there is still strong demand for traditional testing skills.
Ranorex survey respondents were asked, “which languages do the software testers on your team need to know, if any?” The top responses were C# at 62%, followed by SQL at 47% and XML at 46%.
While Ranorex Studio supports all testing applications written for all major platforms, Ranorex code modules are written in either C# or VB.NET, which is reflected in the inclusion of these languages in the survey results. Of course, Ranorex Studio also offers fully codeless creation of tests, so testers without programming skills can build automated tests.
As software testers work more closely with developers, having programming skills is an asset. Therefore, it seems surprising that a review of job advertisements in 2017 showed a decrease in the number of employers requiring programming skills for testers, from 60% in 2010 to 37% in 2017. However, this decrease in requirements for programming experience occurred at the same time as an even greater increase in requirements for test automation experience. It is likely that these two changes are related.
In an industry that is rapidly moving toward continuous testing:
Wondering how to get started with test automation? Ranorex can help.
With Ranorex, test automation is available to coders and non-coders alike. If you are already using Ranorex Studio, you know that our matchless object recognition features and codeless scripting make test automation accessible to non-coders. With only some C# or VB.NET coding skills, testers can customize automation scripts in Ranorex, while those with full C# or VB.NET coding skills can create fully-customized automation scripts. Ranorex also integrates with essential web technologies including Selenium WebDriver and Selenium Grid, HTML5, Adobe Flash and Flex, and supports testing of web applications written in languages such as Java and Javascript. In addition, Ranorex offers a SQL data connector that allows you to build SQL queries and preview their results.
Download a free trial version of Ranorex today to experience the power of test automation. Or, register for one of our free, informative webinars to learn more.
Employment Research Methodology
As in the original 2010 research by Hendrickson, Ranorex used only postings from direct hire jobs, screening out jobs from placement firms and those without an identifiable employer. All postings were accepting applications online between 1 August 2017 and 31 August 2017, and were retrieved from a variety of websites including LinkedIn, Indeed.com, Monster.com and Dice.com. 69% of positions were for placement in the USA, 32% in Europe and 8% in other locations worldwide.Care was taken to eliminate duplicate postings that appeared on multiple sites. Jobs that included the word “test” but primarily seemed to be programming were also excluded. No attempt was made to search for or exclude “Test Automation” positions. However, these types of positions occasionally appeared in searches using keywords such as “Software Tester” and “Software QA.” Of the total number of jobs analyzed for these keywords, 6% were primarily test automation. Jobs were treated as requiring programming experience if they stated that a specific language was required, or if they stated that programming was required without identifying a specific language. However, a request for experience in testing systems written in a specific language was not treated as a requirement for the ability to program in that language.
Contact our Ranorex Studio support team by email. We’ll get back to you as soon as possible.
