When asked about what things are important to track when testing, I find most people give a pretty detailed list of items. They’ll mention system or machine states, DB or other data system availability, structure or readiness. They might point out things like successful builds, available devices and tools, server and network readiness. They could point out mobile devices and emulators devices used by their customers to access their app.
Even when test teams can “control” their environment, there are some things they cannot control. Depending on the nature of the software and organization, teams may be able to control certain features on their test devices, laptops, desktops, mobile devices, emulators and the like. The challenge I sometimes see is if they can control all the features on all their devices?
Again, that will vary from organization to organization. Sometimes, from team to team.
There is the question of the software under test. Development teams are making fixes and deploying them to some environment. Are the testers aware of when this happens? The question of controlling when the new builds get deployed to test environments where testing and what kind of notification occurs is extremely important. This is particularly true when the test environment is shared across multiple teams.
If there is more than one project running in a single environment, make sure the question of changes and possible impacts to each team are made fully public. There needs to be a clear notion of how system and environmental variables are changed. It might be that any testers from any of the teams can make these changes. It may also be some admin not known to you.
What about switches that might be enabled at the database level, or in partitions of the test environment? If you can set them to emulate a particular customer setting, for example, how well can you control them to keep someone else from changing them for their own testing?
Look for program switches that impact functionality. It is possible these get changed in the test environment without everyone being aware of the change. Being aware of what the settings do, how they impact your software and other elements of the broader application is crucial to understanding how the software works.
Make sure you know when they are changed and who controls them in your test environment.
It is important for teams to verify what other teams may be working in the same environment. Teams need to communicate as much as possible over when code is being checked into a shared environment to keep from stepping on each other’s work.
Every project is different. Every team and software organization are different.
When asked, most testers would say they have complete control of their test environment. People will tend to believe that until something comes along to shake their beliefs. For example, asking questions around specific functions, system and environment settings and who can change them.
Are there other things they cannot control themselves?
There may be server settings which impact the stacks in the test environments. This can cause no end of consternation if not well managed. Then there are other questions around server stacks, for example:
These are questions I typically did not consider until I ran full-speed into them. Until then, I was certain I “controlled” the test environment I was working in. Then I found I knew less than I thought I did.
I now have a very different understanding of what “control” of environment and environmental variables looks like.
If you are working in a “cloud” or “server-less” environment, perhaps exercising a SaaS model product, these are concepts you might be familiar with. If you are not, it might be a good idea to find out if there are any present and how they can impact the software and your testing.
If you begin to work in such environments, these are questions to definitely think and ask about.
These are variables we might be able to control. Completely outside of what any test scenario may call for, these combinations of settings and conditions can influence how the software functions.
We have a reasonable chance of controlling them if we know they are there, and how they impact the system.
What has not been considered yet is the level of access testers have to workstation/laptop/desktop settings and server settings. Some questions you might want to consider include:
In many shops I have seen, but by no means all, most testers do. Sometimes, not all. There might be one administrator for certain types of devices beyond computer workstations, tablets and smartphones.
Then there are the server and database control variables. Do testers have the access needed to configure those to match specific conditions and scenarios?
If you have customers, internal or external, do you have the means to configure systems to match how they are using it? What if they are more than one OS upgrade back from “current.” Can you test against all mobile device versions that are supported?
If we can’t control these things, which if we think about it might be kind of obvious, what else might we have a problem with “controlling”?
There is one thing we can’t control, any of us: time. Things will happen whether we are aware of them or not. They may be happening all the time and we don’t notice.
For example, do you know what the “automatic” updates on workstations and servers do before they run? Do you have any control over disabling them? When do you find out about them?
One thing you might want to think about is after a day or two executing complex test scenarios, the workstation or server might get an “automatic” update. Can this impact our tests? We like to think not. Still, if it does, what happens if system settings are changed by default in the upgrade?
What happens to the quality of testing as the deadline for the project or release approaches? The team might launch into overdrive and do excellent work. Of course, the team might put in massive hours and rush through a minimum number of steps to “check the box” that a feature has been tested.
Unless you have a time turner it is very likely that you are not able to do any form of time travel. This limits the amount of “last minute testing” you and the team can do. The problem, of course, is with rushed work, the odds of problems sneaking past you tends to increase.
If the tester is concerned about what is left to do, how much (or little) time there is to do it, and is not focusing attention on what is happening right now, it is likely that small, but potentially important problems will not be noticed.
The view of the testers tends to get pulled to other areas and concerns that the software at hand. Is there a way to avoid this? There may be, but that involves much more than “test faster” or “use more automation” or “deliver the code sooner.”
Even when we think we can control something, it appears we still cannot control when testing can start or finish.
There are other things that can happen that are beyond our control or expectation.
I remember one instance where a group of colleagues were heading out for after-work drinks on a Friday. I had been doing some complex configuration changes on the test system and decided to simply leave them in place until Monday. That would save me massive time reconfiguring the system.
When I came in that Monday, the system was down. There was a memory leak in the application I was testing. By limiting the amount of available memory on the virtual server it was running on and allowing the application to simply idle all weekend, the leak revealed itself by bringing down the server.
While I had not planned on doing that “test” it found an important problem I likely would not have found. It might be wise to consider other things related to time. For example::
Consider what might impact the software and what happens all the time? What if you open a new browser window the next morning, without going back to the original window? What happens if you start multiple sessions and let them all simply idle?
Do you know what might happen if the host/server does something unusual while these multiple sessions are running? What is the result if they all end abruptly, say in rebooting the workstation?
What about one more weird what-if?
Are the workstations set to detect time zones? What happens if you go back an hour, for example, move from Eastern US Time to Central US Time? What if you move up an hour and go the other direction? Can you force the system clock on the workstation to change?
The ideas, questions and concerns raised are only a starting point to think about. Each organization will have a different set of challenges. What else can you not control that you have not thought of trying to control?
There are more and more Android testing tools available for mobile app developers. These are our favorites for performance, accessibility, and security.
Regression testing and retesting are essential methodologies testers employ to ensure software quality. If you’re new to both or aren’t sure when to use which technique, this article should help. We’ll discuss the importance of regression testing in software testing. ...
DevOps test automation enables teams to work faster and create better products. These best practices and tools make it easier to implement test automation.
