Multi-Phase GUI Testing with Ranorex under Cloud-Based CI/CD Part 2: Staging

Modern UI testing is more than firing up an application’s website and then entering data and recording results. Today, a distributed application will be tested in a variety of release stages — from a developer’s local machine, onto a number of progressive external release environments. Such is the nature of Continuous Integration/Continuous Deployment (CI/CD).
Making it so automated UI testing can run smoothly in a CI/CD process that traverses a variety of build environments is a key requirement of the software development lifecycle in the modern enterprise.

This three-part series describes how to deploy and run GUI automation against different release environments. The first article in this series covers using Jenkins to deploy web application source code from GitHub to a build server and then perform a GUI test on the application using Ranorex’s web testing features. In this article, you’ll learn how to use Jenkins to deploy the code to a staging environment in which the web code runs in a container. We’ll use Ranorex’s web testing features to execute tests against the container. In Parts 3 and 4, we’ll deploy the containerized web application to the cloud and run Ranorex web tests against the application running under the Kubernetes cloud-native environment.

Figure 1 below shows the deployment architectures for all three stages:

Understanding the Staging Runway Architecture

The Staging runway is the deployment path for code that is checked into a project’s staging branch on GitHub. The purpose of the Staging runway is to provide a testing environment in which code that has been promoted from the development branch to the staging branch in GitHub can be subjected to functional and integration testing.

The way the Staging build process works is that once the code is promoted from the development branch to the staging branch in GitHub, the Jenkins build server takes the code from GitHub and sends it to a Windows slave server where the sample application is encapsulated into a Docker container. The containerized application runs as a web application on the Windows slave server. The web application is then tested using a Ranorex Studio web test. (See Figure 2)

The reason the code needs to run on a Windows server is due to the fact that Ranorex is a Windows product. The Demonstration Application is written in Node.js and is agnostic in terms of the operating system. But, we need to run on Windows to perform the UI testing using Ranorex.

The following sections show you how to create the Windows Slave server bound to a Jenkins server running on Linux. Also, you’ll learn how to create the Jenkins job that will get the code from GitHub, create the web application in a container using Docker, test it using Ranorex and then publish the test results in the Jenkins UI.

Understanding the Demonstration Application

The code we’re going to deploy and test is a trivial Node.js application that is stored on GitHub. We’re going to conduct a Ranorex Web UI Test that exercises the application’s login feature, The UI test enters user name and password data. Then it clicks the login button and verifies that the expected greeting message appears. Figure 3 below shows the demo application that’s to be tested and the Ranorex UI test that will be conducted.

Jenkins Configuration Prerequisites

Before we go into the steps required to create the Jenkins project that runs the Ranorex UI test on a Windows server, we need to make sure that the Jenkins master server has the necessary plugins installed and the Windows server has the applications needed to build and test the demonstration application.

Jenkins MSBuild Plugin

The Jenkins server requires that MSBuild Plugin is installed in order to execute the build of the Ranorex .NET test solution. The MSBuild Plugin needs to be configured with a unique label that we’ll use to identify it as the plugin to use when compiling the Ranorex test. (See Figure 4)

Jenkins Ranorex Plugin

The Jenkins server also requires that the Ranorex Plugin is installed. This plugin will run the test on the Windows server.

Jenkins Post Build Task Plugin

The Jenkins Post Build Task Plugin must be installed so that Jenkins can run cleanup tasks on the Windows Slave server after the build completes. This plugin will execute cleanup tasks regardless of whether the build succeeds or fails.

The Windows Configuration Prerequisites

The Windows Server requires that the following applications are installed directly on the host system:

• Java
• A licensed copy of Ranorex Studio
• The Chrome browser
• Docker

Configuring the Jenkins Server with a Windows Slave

In order to run a Ranorex Studio test under the Jenkins server, we need to connect a Windows server as a slave to the Jenkins master. Once connected, Jenkins will execute the project’s build tasks on the Windows server.

Conceptually, the way the Jenkins master to Windows slave works is that we’ll configure Jenkins to be aware of the Windows server and know the location of particulars in the Windows server file system. Also, we’ll make it so the Windows slave facilitates the connection back to the Jenkins server using credential information provided by the Jenkins master server. Configuring the Windows slave back to the Jenkins master occurs on the Windows Server. Figure 5 provides a visual description of the Jenkins master to Windows Server binding interaction.

Before we can have Jenkins connect to the Windows Server, we must enable Windows to connect to Jenkins using Web Launch. This is done under Configure Global Security under the Jenkins server as is described in the next sections.

Enabling Slave Binding Using Java Web Start

From the right side of the Jenkins Server, click on Manage Jenkins to display the Manage Jenkins web page. Click Configure Global Security as shown below in Figure 6.

Once in the Configure Global Security page, go down the page to the section Agents/TCP port for JNLP agents. Select the option Random as shown in Figure 7, below. This action will enable Web Launch to bind the Windows Server to the Jenkins master server.

Now it’s time to connect the Windows server to the Jenkins master.

Creating the Windows Slave

Go back to the Manage Jenkins page and select Manage Nodes as shown in Figure 8, below.

In the Manage Nodes section, select New Node from the left side menu, as shown in Figure 9.

When you select New Node, you will be presented with the dialog shown below in Figure 10.

Give the node a Name in the textbox. In this case, we gave the node the name, WindowsSlave.

In the textbox labeled, Remote root directory, enter the location on the Windows server where Jenkins will put its workspace folder and files. Make sure the directory exists on the Windows server.

Give the node a label by entering a value in the Labels text box. In this case, we entered the label, WinBuildServer. The label is very important. We’ll use it to tell Jenkins where to execute the tasks associated with running the Ranorex test on the Windows slave.

Select Launch agent via Java Web Start from the Launch method drop down.

If you like, you can enter a value in the text box labeled, Custom WorkDir path. Jenkins puts its runtime log files in this directory if told to do so.

Save the settings. You’ll be brought back to the Nodes page. Notice that the newly created node, WindowsSlave has a red X on the computer icon. This is because the external slave has not been bound to Jenkins master. (See Figure 11)

Now it’s time to go to the Windows slave. Go to the Windows machine and bring up Jenkins server web page under a browser on the Windows machine. Go to the Configure Nodes page on Jenkins web page. (See Figure 12)

You’ll notice that the Jenkins web page is displaying the information needed to bind to the Jenkins master server. Jenkins is smart enough to know that the Windows slave needs to be configured to the master and show the UI accordingly. You’ll see information about how to access the agent.jar file as well as the text for the connection command for binding to the master.

Now we need to take 4 steps.

Callout 1 in Figure 12 above shows a link to agent.jar. Download the jar file to your local machine and store it in C:\Jenkins\ directory as defined previously when defining the Windows server.

Open an empty text file in your favorite editor and copy the information provided on the Jenkins web page at callout 2 into a text editor on the Windows machine. Save the text as a .bat file (callout 3) in the C:\Jenkins\directory, as shown in callout 4.

Click on the .bat file to execute the connection command. You’ll see connection binding activity as output in a command window as shown below in Figure 13.

Setting Up the Jenkins Job

We’re going to set up the Ranorex test project as a Jenkins Freestyle project. Go to the Jenkins main page and select New Item from the left side menu, as shown below in Figure 15.

You will be presented with the Item Setup Page. Enter the name of the project, in this case, FeelingTrackerStagingBuild. The name is arbitrary. You can choose one that better fits your needs.

Click Freestyle project, then navigate to the bottom of the web page and click the OK button. The project creation process starts. (See Figure 16)

The project configuration page appears as shown below in Figure 17. We’ll add some description text in the Description text area. Navigate to the bottom of the General tab to the checkbox, Restrict where this project can be run.

Check the checkbox as shown above in Figure 17. Also, in the Label Expression, enter WebBuildServer, which is the label we gave the Windows slave instance we created earlier (See Figure 10). Setting the label tells Jenkins to run this build on the Windows Slave.

Next, we need to tell Jenkins the location of the demo application source code on GitHub. (The code on GitHub also contains the files for the Ranorex Studio test.)

Binding to the Development Branch on Git

Navigate to the Source Code Management section of the project page. Select the Git option as shown below in Figure 18.

Enter the URL for the demo application source code on GitHub. Also, set the Branch Specifier to */staging. Remember, the purpose of the exercise is to execute a Ranorex test according to a particular release stage. In this case, the release stage we’re focusing on is Staging. In the Staging stage, we’ll run the Ranorex test directly against the application running in a Docker container using the source code downloaded from the staging branch in GitHub. Later, when we create a Jenkins project to accommodate the build further along at the UAT release stage, we’ll get the code from the /master branch on GitHub.

After we let Jenkins know where to find the source code in GitHub, we need to tell Jenkins how to be aware when new source code is available to test.

Setting Up the Build Trigger

The way we let Jenkins know that there is new code that has been committed to the development branch in GitHub is by setting a Build Trigger.

Navigate down the project configuration page to the section, Build Triggers as shown in Figure 19. Select Poll SCM and enter H/15 * * * * in the Schedule text area.

Setting the schedule to H/15 * * * * tells Jenkins to check GitHub every 15 minutes to see if there is new code to download and build.

Having told Jenkins how and when to get source code from GitHub, we need to build it.

Containerizing the Node Application Using Docker

Navigate down the Jenkins project setup page to the Build section (See Figure 20). From the Add build step dropdown, select Execute Windows batch command. We need to use a Windows batch command because this job will be running on the Windows slave server. This is a very important distinction. Remember, the Jenkins server is running on a Linux machine, but the Ranorex test will execute on a Windows machine. Thus, we need to execute command line instructions that are compatible with Windows.

In this case, we’re going to tell Jenkins to encapsulate the Node.js demo app into a Docker container. The encapsulation will happen on the Windows server using a command line instruction we’ll enter into the Windows batch command text area. In the Command text area enter the text:

docker build -t feelingtracker:v1 .

Running docker build -t feelingtracker:v1 . creates a Docker container image named feelingtracker:v1 using the information provided by the file, Dockerfile. Dockerfile is part of the source code Jenkins downloaded from the /staging branch in the demo app’s GitHub repository. The presence of the file, Dockerfile is required by Docker. Using Dockerfile is a standard part of the Docker process for building a container image. Once an image is created, we’ll execute docker run to create a container based on the image.

Next, we’ll add another Windows batch command task in Jenkins into which we’ll enter the code to create and run the container that contains the demo app.

In the text area of the newly added task enter the command:

docker run -d --name myapp_container -p 3000:3000 -e DEPLOYMENT_ENV=STAGING -e CURRENT_VERSION=v1 feelingtracker:v1

See Figure 21 below.

The docker run command creates a container named myapp_container. The container exposes port 3000 so that the demo application in the container is accessible within the Windows Server. Exposing port 3000 allows Ranorex to access the demo app in a browser at the URL, localhost:3000. Also, docker run injects two environment variables into the running Docker container. One environment variable, DEPLOYMENT_ENV describes STAGING as the current deployment environment. The other environment variable, CURRENT_VERSION describes the current version of the application. The current version is set to v1. These environment variables are arbitrary to the demonstration application. They are in no way required by Docker.

Once we have the demonstration application up and running in a container, we need to test it. First, we need to build the Ranorex Studio test suite.

Building the Ranorex Studio Test Suite

We’re going to build the Ranorex test suite under a new build step. You’ll see an Add build step drop-down button beneath the previous step we built. Click that button and select, Build a Visual Studio project or solution using MSBuild, as shown below in Figure 22.

In the drop-down labeled MSBuild Version, select the name of the MSBuild plugin you installed on the Jenkins server. In this case, the Jenkins server has an MSBuild plugin named ReselbobPCBuild.

In the MSBuild Build File text box, enter the path to the Ranorex solution file (.sln) that will drive the build compilation. In this case, the entire Ranorex build solution is stored in the GitHub project, in the directory \ranorex\feelingTrackerRanorexTest. (See Figure 23)

When Jenkins retrieves the demo application project files from GitHub, it puts them in the Jenkins project workspace. Thus, we set the location of the Ranorex solution file (.sln) relative to the Jenkins workspace. Jenkins will run MSBuild against the solution file. The executable that is the result of the build will be used to fire off the Ranorex test file in an upcoming build step.

Now we need to start the demonstration application.

Running the Ranorex Studio test case

Add a new build step to the Jenkins project. But this time, select Run a Ranorex test suite, from the build steps drop-down menu, as shown below in Figure 24. (This assumes you have installed the Ranorex plugin previously, as required.)

The way the Ranorex plugin works is that you run the .NET test executable compiled earlier against a .rxtst test file. Thus, in the Ranorex test suite file textbox, we enter the relative path to the file feelingTrackerRanorexTest.rxtst that was copied into the ..\bin\Debug\ directory of the Ranorex test solution folder by MSBuild during the .NET compilation process.

Enter a report name in the textbox, Ranorex report file name. Also, click the checkbox for JUnit-compatible report, as shown above in Figure 24.

This step will execute the Ranorex test on the Windows server and generate an XML file, BasicReport.rxlog.junit.xml which describe the results of the Ranorex test. We’ll use this file in a later step.

Cleaning Up After the Build

After the test has run, we need to remove from the Windows Server both the running container and the Docker image that was used to build the container. We’re going to accomplish these tasks by using a Jenkins post-build task.

Navigate to the end of the Jenkins job we’ve built and click the button Add post-build action as shown below in Figure 24 (callout 1). Then select, Post build task from the drop-down list (callout 2).

Enter the text, \. in the Log text textbox of the Post-build task dialog. The purpose of the Log text textbox is to provide a way to search the log file that is generated automatically by Jenkins during the build to identify text that indicates a condition that warrants executing the Post-build task.

The text, \. is a Java regular expression that indicates any character (See Figure 26, callout 1). Jenkins will apply the regular expression to the log file text that is generated during the build. Operationally we’re telling Jenkins to go into the log file and identify any character at all. Once any character is identified, execute the Post-build task. Effectively we’re having Jenkins execute the Post-build task under any condition, regardless of whether the build was successful or failed. However, there might be other situations later on in which you’ll want to execute a Post-build task under a certain condition, for example, a FAILURE. In a case such as this, you’ll create regular expression entry in Log text textbox that identifies log output that indicates the type of failure you’re interested in.

For this task we want the container to stop running under any condition, pass or fail. Thus, after entering the “any character” regular expression, we enter the text docker stop myapp_container in the Script textbox as shown below in Figure 26, callout 2.

Once the container is stopped, it can be deleted. Add another Post build task to the build and enter the regular expression text \. into the Log text textbox. (See Figure 27, callout 1, below). Enter the text, docker rm myapp_container in the Script textbox as shown in Figure 27 (callout 2).

The command, docker rm myapp_container removes the container from the Windows Server. Next, we need to remove the Docker image that was used to create the container.

Add another Post build task to the build and enter the regular expression text, \. into the Log text textbox. (See Figure 28, callout 1, below). Remember, we need to enter the regular expression for “any character” in order to have the Post-build task execute regardless of success or failure.

Enter the Docker command, docker image rm feelingtracker:v1 in the Script textbox as shown below in Figure 28, callout 2.)

This Post build task removes the container image from the Windows Server. All remnants of the containerized demo app have been removed from the Window Server. This means that subsequent test runs will execute in a Windows Server environment that is completely clean.

Next, we need to add another post-build action that reports the results of the Ranorex test.

Defining Reports as Post Build Task

Click the Post-build Actions button and select, Publish JUnit test result report. (See Figure 29)

In the Test reports XML text area, enter the path to the test XML file we described earlier in Figure 23 when configuring the Ranorex test suite build step. Notice that the XML file will be stored in the ..\bin\Debug directory on the Windows slave server. This directory is where Ranorex builds the test .exe and runs the test. Also, make sure that the name of the test report matches the report name you defined when configuring the Ranorex testing step earlier. If the names do not match up, Jenkins will display an error message in the Post-build section reporting that it can’t find the test file.

Setting up the build is complete. Click the Save button at the bottom of this build configuration page. Now it’s time to run the build.

To run the build, click the link Build Now on the left side menu of the Web page of the build project page, as shown below in Figure 30.

The build process begins. After the testing completes, the build process will shut down all running instances of Node.js and Chrome on the Windows slave server and publish the test reports. The Jenkins UI will display a Test Results Trend chart that reports test results over many builds.

Next Steps

In this article, we looked at how to run a Ranorex test in the Staging runway of the release process. We ran the tests on a Windows slave bound to the master Jenkins server.

The next step in the multi-phase GUI testing process is to go to the UAT release phase. In this phase, we’ll create a new Jenkins build project that gets the demonstration application code from the Master branch of the GitHub repository. The Jenkins build process for the UAT phase will send the containerized demonstration application to a Kubernetes cluster running on Google Cloud and then use Ranorex to test the demonstration application’s UI as it’s running in the Kubernetes Cluster.