Modern UI testing is more than firing up an application’s website and then entering data and recording results. Today, a distributed application will be tested in a variety of release stages — from a developer’s local machine, onto a number of progressive external release environments. Such is the nature of Continuous Integration/Continuous Deployment (CI/CD).
Making it so automated UI testing can run smoothly in a CI/CD process that traverses a variety of build environments, is a key requirement of the software development lifecycle in the modern enterprise.
This three-part series describes how to deploy and run GUI automation against different release environments. This first part will cover using Jenkins to deploy web application source code from GitHub to a build server and then perform a GUI test on the application using Ranorex’s web testing features.
The second part will show how to use Jenkins to deploy the code to a staging environment in which the web code runs in a container. We’ll use Ranorex’s web testing features to execute tests against the container.
Finally, in Part 3, we’ll deploy the containerized web application to the cloud and run Ranorex web tests against the application running under Kubernetes cloud-native environment.
Figure 1 below shows the deployment architectures for all three stages.
Figure 1: A Multi-Phase GUI Testing Architecture using Ranorex, Jenkins, Docker and Kubernetes
Let’s get started with Part 1, deploying and building source code in a Development environment and then running Ranorex web tests against the running web application in the environment.
Understanding the Development Runway Architecture
The Development runway is the deployment path for code that is checked into a project’s development branch on GitHub. The purpose of the Development runway is to provide an application environment in which code from all the developers is unified into a single code base that can be reviewed and tested.
The way the Development build process works is that once the developer checks code into the GitHub repository’s development branch from his her local machine, the Jenkins build server takes the code from GitHub and sends it to a Windows slave server where the sample application is built, started and tested using a Ranorex web test. (See Figure 2.)
Figure 2: The deployment architecture for the Development runway
The reason the code needs to run on a Windows server is due to the fact that Ranorex is a Windows product. The Demonstration Application is written in Node.js and is agnostic in terms of Operating System. But, we need to run on Windows to perform the UI testing using Ranorex.
The following sections show you how to create the Windows Slave server bound to a Jenkins server running on Linux. Also, you’ll learn how to create the Jenkins job that will get the code from GitHub, test it using Ranorex and then publish the test results in the Jenkins UI.
Understanding the Demonstration Application
The code we are going to deploy and test is a trivial Node.js application that is stored on GitHub. We’re going to conduct a Ranorex Web UI Test that exercises the application’s login feature, The UI test enters username and password data. Then it clicks the login button and verifies that the expected greeting message appears. Figure 3 below shows the demo application that’s to be tested and the Ranorex UI test that will be conducted.
Figure 3: The demo application and Ranorex test to be executed
The Ranorex test described above will execute on a Windows slave server bound to Jenkins.
Before we go into the steps required to create the Jenkins project to run Ranorex UI test on a Windows server, we need to make sure that the Jenkins master server has the necessary plugins installed and the Windows server has the applications needed.
Jenkins Configuration Prerequisites
The Jenkins server requires that MSBuild Plugin is installed in order to execute the build of the Ranorex .NET test solution.
The Jenkins server also requires that the Ranorex Plugin is installed. This plugin will run the test on the Windows server.
The Windows Configuration Prerequisites
The Windows Server requires that the following applications are installed directly on the host system:
Configuring the Jenkins Server with a Windows Slave
In order to run a Ranorex test under the Jenkins server, we need to connect a Windows server as a slave to the Jenkins master. Once connected, Jenkins will execute the project’s build tasks on the Windows server.
Conceptually, the way the Jenkins master to Windows slave works is that we’ll configure Jenkins to be aware of the Windows server and know the location of particulars in the Windows server file system. Also, we’ll make it so the Windows slave facilitates the connection back to the Jenkins server using credential information provided by the Jenkins master server. Configuring the Windows slave back to the Jenkins master occurs on the Windows Server. Figure 4 provides a visual description of the Jenkins master to Windows Server binding interaction.
Figure 4: The Jenkins master to Windows Server binding process involves having both systems aware of each other
Before we can have Jenkins connect to the Windows Server, we must enable Windows to connect to Jenkins using Web Launch. This is done under Configure Global Security under the Jenkins server as is described in the next sections.
Enabling Slave Binding Using Java Web Start
From the right side of the Jenkins Server, click on Manage Jenkins to display the Manage Jenkins web page. Click Configure Global Security as shown below in Figure 5.
Figure 5: Select Global Security to access the page to enable Web Launch from the Windows Slave
Once in the Configure Global Security page, go down the page to the section Agents/TCP port for JNLP agents. Select the option Random as shown in Figure 6, below. This action will enable Web Launch to bind the Windows Server to the Jenkins master server.
Figure 6: Select the Random option to enable the Windows Server to use Web Launch to connect to the Jenkins Master
Now it’s time to connect the Windows server to the Jenkins master.
Creating the Windows Slave
Go back to the Manage Jenkins page and select Manage Nodes as shown in Figure 7, below.
Figure 7: The Windows Server will be connected to the Jenkins master as a new node
In the Manage Nodes section, select New Node from the left side menu, as shown in Figure 8.
Figure 8: Select New Node in the Manage Nodes page to start the process to create a Windows slave
When you select New Node, you will be presented with the dialog shown below in Figure 9.
Figure 9: The page for configuring a Windows slave
Give the node a Name in the textbox. In this case. we gave the node the name, Windows Slave.
In the textbox labeled Remote root directory, enter the location on the Windows server where Jenkins will put its workspace folder and files. Make sure the directory exists on the Windows server.
Give the node a label by entering a value in the Labels textbox. In this case, we entered the label, WinBuildServer. The label is very important. We’ll use it to tell Jenkins where to execute the tasks associated with running the Ranorex test on the Windows slave.
Select Launch agent via Java Web Start from the Launch method drop-down.
If you like, you can enter a value in the text box labeled Custom WorkDir path. Jenkins puts its runtime log files in this directory if told to do so.
Save the settings. You’ll be brought back to the Nodes page. Notice that the newly created node, WindowsSlave has a red X on the computer icon. This is because the external slave has not been bound to Jenkins master. (See Figure 10.)
Figure 10: A listing for an external node that still needs to be connected to the master
Now it’s time to go to the Windows slave. Go to the Windows machine and bring up Jenkins server web page under a browser on the Windows machine. Go to the Configure Nodes page on Jenkins web page. (See Figure 11.)
You’ll notice that the Jenkins web page is displaying the information needed to bind to the Jenkins master server. Jenkins is smart enough to know that the Windows slave needs to be configured to the master and show the UI accordingly. You’ll see information about how to access the agent.jar file as well as the text for the connection command for binding to the master.
Now we need to take 4 steps.
Figure 11: The Jenkins web page accessed from the Windows server provides access to the agent.jar file and connection command for connecting to the master
Callout 1 in Figure 11 above shows a link to agent.jar. Download the jar file to your local machine and store it in C:\Jenkins\ directory as defined previously when defining the Windows server.
Open an empty text file in your favorite editor and copy the information provided on the Jenkins web page at Callout 2 into a text editor on the Windows machine. Save the text as a .bat file (Callout 3) in the C:\Jenkins\ directory, as shown in Callout 4.
Click on the .bat file to execute the connection command. You’ll see connection binding activity as output in a command window as shown below in Figure 12.
Figure 12: Executing the command displayed on the Jenkins web page binds the Windows server to the Jenkins master
Go back to the Configure Nodes web page on the Jenkins master after the connection command executes. You’ll notice that the external Windows slave is connected and ready for use. (See Figure 13)
Figure 13: A Windows slave configured as a node on the Jenkins server
Now we’re ready to set up the Jenkins job to run the Ranorex test on the Windows Slave.
Setting Up the Jenkins Job
We’re going to setup the Ranorex test project as a Jenkins Freestyle project. Go to the Jenkins main page and select New Item from the left side menu, as shown below in Figure 14.
Figure 14: Clicking New Item shows the project selection page
You will be presented with the Item Setup Page. Enter the name of the project, in this case, FeelingTrackerDevBuild. The name is arbitrary. You can choose one that better fits your needs.
Click on Freestyle project, then navigate to the bottom of the web page and click the OK button. The project creation process starts. (See Figure 15)
Figure 15: Set up the Ranorex test project as a Jenkins Freestyle Project
The project configuration page appears as shown below in Figure 16. We’ll add some description text in the Description text area.
Navigate to the bottom of the General tab to the checkbox Restrict, where this project can be run.
Figure 16: Checking ‘Restrict where this project can be run’ and setting the Windows server label defined earlier tells Jenkins to run the build tasks on the Windows slave
Check the checkbox as shown above in Figure 16. Also, in the Label Expression, enter WebBuildServer, which is the label we gave the Windows slave instance we created earlier. (See Figure 9.) Setting the label tells Jenkins to run this build on the Windows Slave.
Next, we need to tell Jenkins the location of the demo application source code on GitHub. (The code on GitHub also contains the files for the Ranorex test.)
Binding to the Development Branch on Git
Navigate to the Source Code Management section of the project page. Select the Git option as shown below in Figure 17.
Figure 17: The Jenkins task to define location and branch of the demonstration application source code in GitHub
Enter the URL for the demo application source on GitHub. Also, set the Branch Specifier to */development. Remember, the purpose of the exercise is to execute a Ranorex test according to a particular release stage. In this case, the release stage we’re focusing on is Development. In the Development stage, we’ll run the Ranorex test directly against the application running from the source code downloaded from the development branch in GitHub. Later, as create Jenkins projects to accommodate builds further along in the Staging and UAT release stages, we’ll get the code from the /staging and /master branch on GitHub respectively.
After we let Jenkins know where to find the source code in GitHub, we need to tell Jenkins how to be aware when new source code is available to test.
Setting Up the Build Trigger
The way we let Jenkins know that there is new code has been committed to the development branch in GitHub is by setting a Build Trigger.
Navigate down the project configuration page to the section, Build Triggers as shown in Figure 18. Select Poll SCM and enter H/15 * * * * in the Schedule text area.
Figure 18: The Jenkins task for monitoring the demonstration application source in GitHub for changes to checkout
Setting the schedule to H/15 * * * * tells Jenkins to check GitHub every 15 minutes to see if there is new code to download and build.
Having told Jenkins how and when to get the source code from GitHub, we now need to build it.
Building the Node Application
Navigate down the Jenkins project setup page to the Build section. (See Figure 19.) From the Add build step dropdown, select Execute Windows batch command. We need to use a Windows batch command because this job will be running on the Windows slave server. This is a very important distinction. Remember, the Jenkins server is running on a Linux machine, but the Ranorex test will execute on a Windows machine. Thus, we need to execute command line instructions that are compatible with Windows.
In this case, we’re going to tell Jenkins to build the Node.js demo app on the Windows server using the command line instruction. In the Command text area enter the text npm install.
Figure 19: The Windows command to build the Node.js demonstration application
Running npm install downloads all the dependency packages from the internet that the demo application requires. Next, we need to compile the Ranorex test.
Building the Ranorex Test Suite
We’re going to build the Ranorex test suite under a new build step. You’ll see an Add build step drop-down button beneath the previous step we built. Click that button and select, Build a Visual Studio project or solution using MSBuild, as shown below in Figure 20.
Figure 20: The MSBuild command to compile the Ranorex test
In the drop-down labeled MSBuild Version, select the name of the MSBuild plugin you installed on the Jenkins server. In this case, the Jenkins server has an MSBuild plugin named ReselbobPCBuild.
In the MSBuild Build File text box, enter the path to the Ranorex solution file (.sln) that will drive the build compilation. In this case, the entire Ranorex build solution is stored in the GitHub project, in the directory \ranorex\feelingTrackerRanorexTest. (See Figure 21)
Figure 21: The Ranorex test solution is stored along with the demo application source code on GitHub
When Jenkins retrieves the demo application project files from GitHub, it puts them in the Jenkins project workspace. Thus, we set the location of the Ranorex solution file (.sln) relative to the Jenkins workspace. Jenkins will run MSBuild against the solution file. The executable that is the result of the build will be used to fire off the Ranorex test file in an upcoming build step.
Now we need to start the demonstration application.
Starting the Node Application
Add another Windows batch command to the Jenkins build process, as we did previously in Figure 19. This command will start the demonstration app. In the Command textarea enter: forever start app.js (See Figure 22. Remember, having the forever Node package installed on the Windows server is a configuration prerequisite for running this Jenkins project.)
Figure 22: The Windows command to start the Demo application as a background process
The forever command starts the demo application in a separate process on the Windows server. If we did not run the demo application in a separate process, the demo application’s running web server would block out the rest of the Jenkins build tasks. The Jenkins build would appear to hang. Running the demo application in a separate process allows the Jenkins build to continue.
Now that the demo application is up and running we can conduct the Ranorex test.
Running the Ranorex Test Case
Add a new build step to the Jenkins project. But this time, select Run a Ranorex test suite, from the build steps drop-down menu, as shown below in Figure 23. (This assumes you have installed the Ranorex plugin previously, as required.)
Figure 23: The build task to execute running the Ranorex test on the Windows server
The way the Ranorex plugin works is that you run the .NET test executable compiled earlier against a .rxtst test file. Thus, in the Ranorex test suite file text box, we enter the relative path to the file feelingTrackerRanorexTest.rxtst that was copied into the ..\bin\Debug\ directory of the Ranorex test solution folder by MSBuild during the .NET compilation process.
Enter a report name in the textbox, Ranorex report file name. Also, click the checkbox for JUnit-compatible report, as shown above in Figure 23.
This step will execute the Ranorex test on the Windows server and generate an XML file, BasicReport.rxlog.junit.xml, which describes the results of the Ranorex test. We’ll use this file in a later step.
Cleaning Up After the Build
After the test has run, we need to shut down all instances of Node.js and Chrome running on the Windows server. We need to do this so that subsequent test runs on the Windows server are not subject to preexisting conditions that create errors. For example, trying to start the demo application’s web server will error, if the application is already running.
To shut down all instances of Node.js running on the Windows server, add a new Windows command build step, and in the Command text area, enter taskkill /im node.exe /F, as shown in Figure 24 below.
Figure 24: Executing the command the terminates all running instance of Node.js on the Windows server
Next, add another Windows command build step and enter the command, taskkill /im chrome.exe /F, as shown in Figure 25 below. This command will shut down all running instances of Chrome on the Windows slave.
Figure 25: Executing the command the terminates all running instance of Chrome on the Windows server
The last thing we need to do is to configure Jenkins to display the Ranorex test results in the Jenkins UI.
Defining Reports as Post-Build Task
You’ll see the Post-build Actions section at the bottom of the Jenkins project page. Click the Add post-build action dropdown button and select Publish JUnit test result report. (See Figure 26)
Figure 26: Defining the location the Windows slave where the test report is saved
In the Test reports XML text area, enter the path to the test XML file we described earlier in Figure 23 when configuring the Ranorex test suite build step. Notice that the XML file will be stored in the ..\bin\Debug directory on the Windows slave server. This directory is where Ranorex builds the test .exe and runs the test. Also, make sure that the name of the test report matches the report name you defined when configuring the Ranorex testing step earlier. If the names do not match up, Jenkins will display an error message in the Post-build section reporting that it can’t find the test file.
Setting up the build is complete. Click the Save button at the bottom of this build configuration page. Now it’s time to run the build.
To run the build, click the link Build Now on the left side menu of the Web page of the build project page, as shown below in Figure 27.
Figure 27: The results of many runs of the Jenkins job to execute the Ranorex tests
In this article, we looked at how to run a Ranorex test in the Development runway of the release process. We ran the tests on Windows slave bound to the master Jenkins server.
The next step in the multi-phase GUI testing process is to go to the Staging phase. In this phase, we’ll create a new Jenkins build project that gets the demonstration application code from the Staging branch of the GitHub repository. The Jenkins build process for this stage will be to run the demo application from within a Docker container and make it so the Ranorex UI tests execute against the running container instance.