Accelerating Toward Continuous Testing
9 things to know about software testing on agile teams
1. The transition from waterfall to agile
development is still in progress.
The profession of software testing is undergoing rapid change, driven by a transformation in software development practice. Until recently, the “waterfall” model of software development has been the leading practice. According to Gartner, Inc., “waterfall methods were employed on 56% of development efforts in 2015, with iterative methods used in 21% and agile in 23%.”
Now, agile teams are in the majority. In a recent online survey, just over 80% of professional developers reported using agile methods, compared to 27% using waterfall methods. The proportion of agile development was even higher in an email survey of over 230 testing professionals conducted by Ranorex in August 2017, where 90% of respondents reported following agile practices. However, the transition to agile is incomplete: 8% of respondents to the Ranorex survey describe their development practice as still “primarily waterfall,” while another 35% are using a combination of agile and waterfall practices.
Ranorex Survey: Agile vs Waterfall
Ranorex Survey: Agile vs Waterfall
When do waterfall teams plan to adopt agile practices?
How long ago did survey respondents who are now in agile or mixed environments report working in a primarily waterfall environment?
2. Agile teams follow a wide variety of development practices.
The goal of an agile team is to increase customer satisfaction through rapid delivery of a high-quality solution. Certain practices are common to all forms of agile:
- Close collaboration between all members of a project team, including developers, testers, business representatives, and customers
- Responsiveness to changing requirements
- Frequent releases of working software
Within these broad principles, there is wide variation in how teams practice agile development.
Ranorex Survey: Primary Development Practices
Scrum is the most popular development practice.
Many teams use a combination of practices.
Almost half of teams follow continuous practices.
“We…work in small focused teams following bits of Scrum, bits of Kanban, with a healthy dose of Extreme Programming…. Our cross functional teams have Product Management, all aspects of Product Development (including DevOps and Test), and UX all working together on a daily basis… The team works very closely, and like to pair program on pretty much everything.”
3. Interest in DevOps is growing.
In the Ranorex survey, 11% of respondents reported using DevOps as a primary development practice, and another 18% as a secondary practice. This is similar to the results in the 2017 Stack Overflow survey, where 11% of developers identified themselves as “DevOps specialists;” the same survey reported that DevOps specialists are the highest paid developers worldwide, well ahead of mobile and web developers.
A key indicator of interest in a subject is the number of related web searches. A Google Trends analysis shows that searches for the term ”DevOps” have increased steadily since 2012, reaching their current peak in the summer of 2017. By comparison, interest in the term “Scrum” during the same period remained relatively unchanged.
DevOps takes the principles of Agile and expands their scope, recognizing that ensuring high-quality development requires continual engagement and feedback from a variety of technical experts, including QA and operations specialists.
4. Testing is shifting left.
Primarily agile projects vs. mixed agile/waterfall
The shortest iterations are found in CI/CD and DevOps environments, where new code may become available for testing every day – or even several times per day. Testing in continuous environments presents unique challenges. First, tests must be ready to run as soon development completes, which means that testers might begin writing their tests while developers are still working on the code. Tests may have to be re-written quickly in response to changing requirements. In addition, automated tests must complete in a very short time frame, so it may be necessary to execute them on multiple remote systems, in parallel.
Given these challenges, continuous testing environments benefit from a combination of automated and manual tests, as shown below.
5. Teams benefit when testers collaborate with developers.
Ranorex Survey: QA/Tester Planning Responsibilities
What planning tasks do testers on agile teams do regularly?
Tester collaboration during planning
In the Ranorex survey, respondents on agile teams reported that they commonly create test artifacts such as test cases, participate in the creation of test plans, and help identify areas to test. However, fewer respondents said that they were able to help write testable user requirements or define acceptance criteria. On fully agile teams, these numbers should be closer to 100%.
Ranorex Survey: QA/Tester Project Responsibilities
What project tasks do testers on agile teams do regularly?
Tester collaboration during development
During development, testers and developers on agile teams should work in pairs to ensure that effective unit testing is occurring; and that testers understand the product and its requirements to support exploratory testing. However, only 22% of respondents to the Ranorex survey reported that they assist developers in this way. Just 11% of respondents regularly coach developers in testing techniques. More commonly, testers create automated test cases (54%), conduct exploratory testing (51%) or scripted testing (46%), and analyze completed tests (53%).
Our experience with agile teams has shown that testing skills and experience are vital to project success, and that testers do add value to agile teams.
6. Team organization is important for success.
Respondents to the Ranorex survey reported several benefits of testing in agile environments:
- “Shorter cycles means smaller code, which means the potential of finding and correcting issues faster.”
- “[Allows] more frequent testing – including updating of tests.”
- “Constant/continuous feedback…helps to keep developers on track.”
- “Testing can be done in a timely manner during the sprint; changes can be integrated quickly.”
However, some teams are dissatisfied with their current practices.
35% of respondents to the Ranorex survey are in “mixed agile and waterfall environments.” Testing can be negatively affected when a sprint becomes a mini-waterfall, with activities occurring in shortened but still structured phases. Testers may have nothing to do at the start of the sprint, and then be rushed at the end of the sprint. Or, testing may be delayed until the start of the next sprint. To avoid this issue, system and acceptance testing should occur as each story is completed, rather than for all stories at the end of a sprint – or worse, in a special “testing” sprint, as shown below:
Concerns expressed in the Ranorex survey included:
- Resistance to change when large parts of the organization use waterfall methods
- Testing pushed to the end of a sprint, so there is insufficient time for testing
- A lack of management understanding regarding agile practices
- Waterfall-style separation of roles into ‘QA’ and ‘Dev’, rather than a whole-team approach.
Best practice calls for software testers to be full members of the team.
Many waterfall practices don’t work well in combination with an agile environment. Foremost among these is keeping QA and development as separate groups. For an agile environment to be successful, testers should function as full members of the development team. QA should contribute to sprint planning so that there are adequate resources for testing, participate in customer/business meetings, help develop testable story requirements, and partner closely with developers throughout the iteration window .
In a “three amigos” meeting, a product owner, developer, and tester work together to create testable user stories and acceptance criteria.
7. More employers want test automation experience.
Employment Research: Test Automation Experience
Employment Research: Top Requested Automation Technologies
Employment Research: Top Requested Automated Test Management Tools
8. Traditional approaches to testing are still valued.
Most teams see a need for GUI testing.
The increasing emphasis on unit and integration testing sometimes raises questions about the relative importance of traditional GUI testing. However, the ideal mix of testing for a given project depends on the nature of the application and the level of risk. A mobile app may need a higher volume of automated GUI tests relative to unit and API tests since much of its functionality is in the GUI. In addition, unit and integration testing don’t address important aspects of user experience (UX) and usability testing.
Ranorex Survey: GUI Testing
Software testers on teams that are primarily agile:
Employers are looking for traditional skills.
While 70% of positions asked for automated testing experience, 41% specifically mentioned “manual testing” as one of the testing skills either required or preferred. 74% of positions listed structured testing experience, such as experience in writing and executing test cases or test scripts. These results demonstrate that even as the separation of roles between developer and tester is decreasing, there is still strong demand for traditional testing skills.
Employment Research: Types of Testing Experience Desired or Required
9. Fewer job advertisements are requiring programming skills for testers.
Ranorex survey respondents were asked, “which languages do the software testers on your team need to know, if any?” The top responses were C# at 62%, followed by SQL at 47% and XML at 46%.
Ranorex Survey: Top Languages
Question: which languages do the software testers on your team need to know, if any?
While Ranorex Studio supports all testing applications written for all major platforms, Ranorex code modules are written in either C# or VB.NET, which is reflected in the inclusion of these languages in the survey results. Of course, Ranorex Studio also offers fully codeless creation of tests, so testers without programming skills can build automated tests.
As software testers work more closely with developers, having programming skills is an asset. Therefore, it seems surprising that a review of job advertisements in 2017 showed a decrease in the number of employers requiring programming skills for testers, from 60% in 2010 to 37% in 2017. However, this decrease in requirements for programming experience occurred at the same time as an even greater increase in requirements for test automation experience. It is likely that these two changes are related.
Employment Research: Programming Experience for Testers
In an industry that is rapidly moving toward continuous testing:
- Turnaround times are decreasing.
- Fast feedback is essential.
- While the line between the role of developer and tester is blurring, traditional testing skills are still in demand.
- Requirements for test automation skills are increasing.
Wondering how to get started with test automation? Ranorex can help.
Download a free trial version of Ranorex today to experience the power of test automation. Or, register for one of our free, informative webinars to learn more.
Employment Research Methodology
As in the original 2010 research by Hendrickson, Ranorex used only postings from direct hire jobs, screening out jobs from placement firms and those without an identifiable employer. All postings were accepting applications online between 1 August 2017 and 31 August 2017, and were retrieved from a variety of websites including LinkedIn, Indeed.com, Monster.com and Dice.com. 69% of positions were for placement in the USA, 32% in Europe and 8% in other locations worldwide.Care was taken to eliminate duplicate postings that appeared on multiple sites. Jobs that included the word “test” but primarily seemed to be programming were also excluded. No attempt was made to search for or exclude “Test Automation” positions. However, these types of positions occasionally appeared in searches using keywords such as “Software Tester” and “Software QA.” Of the total number of jobs analyzed for these keywords, 6% were primarily test automation. Jobs were treated as requiring programming experience if they stated that a specific language was required, or if they stated that programming was required without identifying a specific language. However, a request for experience in testing systems written in a specific language was not treated as a requirement for the ability to program in that language.