In this final phase, UAT, the software under test will be containerized as it was in the Staging build. But, instead of running the container directly on the Jenkin’s server, we’re going to deploy the container image to Google Cloud’s Container Registry. Then, a container based on the image in the Container Registry will be deployed to a Kubernetes Cluster running on Google Cloud and exposed to the Internet from there. The GUI tests were created in Ranorex Studio and are designed to run as standalone tests from within a Windows environment. The Ranorex GUI tests will be built and run from a Jenkins Windows slave server running on Azure. Figure 1, below illustrates the overall test architecture for the UAT Phase.
Working with Kubernetes on Google Cloud
What makes this phase of deployment and testing different from the previous testing phases? The previous phases deployed the application directly to a Windows slave machine attached to Jenkins. In this UAT phase, the master Jenkins server running on Linux will create a container image of the demo application using source code downloaded from GitHub. Then, the Jenkins server deploys the container image to Google’s Container Registry. Once the image is in the Google Container Registry, additional code in the Jenkins build creates a Kubernetes deployment using the container image in the Container Registry. In addition, the Jenkins server creates a Kubernetes service that is bound to pods that are part of the Kubernetes deployment.
Hint
You will need to have an account on Google Cloud. You can create a Google Cloud account on this web page. You will need a Gmail email address to complete the signup process.
Creating the Service Account and Downloading Access Credentials
Giving Jenkins access to the Google Cloud requires that we create a service account under the Google Cloud Project dedicated to this build and deploy process. The service account will have the rights to work with the Google Container Registry and the Google Kubernetes Engine. Also, as part of the service account creation process, you’ll download a JSON file that contains the credentials that Jenkins will use to access the Google Cloud resources associated with the build. This JSON file needs to stored in a safe place. You’re going to add this information to Jenkins as a secret credential using the Jenkins Credentials Binding Plugin. To create a service account for the Jenkins build on Google Cloud, perform the following steps: Login into your Google Cloud account and at the top of the Google Console web page, select the project into which you want to add the service account, as shown in callout 1 of Figure 2 below. Then, enter the term, IAM in the search text box as shown in callout 2. Select IAM & admin from the dropdown that appears.
- Click the link ADD ANOTHER ROLE, as shown below in Figure 5
- Select a role from the dropdown that appears.
When all three roles have been defined, click the CONTINUE button.
Creating the Kubernetes Cluster
Go to the Search text box at the top of the Google Cloud Console web page, and do a search on the term,
Using the Google Cloud Container Registry
Sharing container images has become a common practice in modern software development. The standard mechanism for sharing container images is the repository. Repositories can be public or private. The industry standard that has evolved is DockerHub, but there are others. In fact, Google Cloud has a container repository of its own. It’s called the Container Registry. Figure 11, below shows the Container Registry page for the Google Project related to this article.
System Requirements for Running Docker and kubectl on the Jenkins Server
The master Jenkins server needs to have certain programs installed on it in order to support deploying the demo code to Google Cloud. These programs are:
- Docker: The program will create the container image that gets deployed to Google Cloud
- Google Cloud SDK: The SDK contains the command line programs that allow the Jenkins build to interact with Google Cloud
- kubectl: The command line program for interacting with the Kubernetes Cluster operating in Google Cloud.
The following links provide the instructions for installing each requirement on the Jenkins server.
- How to Install Docker on Linux [CentOS, Debian, Fedora, Ubuntu]
- How to Install Google Cloud SDK on Linux
- How to Install Kubectl on Linux
Once the requirements are installed, we need to make it possible for the Jenkins server to access Google Cloud. This will be accomplished by using the Jenkins Credential Plugin and Credential Binding Plugin to store sensitive security information required for access.
Installing and Configuring the Credentials Plugin on Jenkins Master
You install the Credential Plugin and Credentials Binding Plugin using the Jenkins Plugin Manager as shown below in Figure 12:
- Select the Kind as Secret text
- Set the Scope as Global
- Add the text that you want to be saved as a secret in the Secret textbox (callout 2)
- Give the secret a unique ID (callout 3)
- Click OK to save the credential (callout 4)
| Name | Contents |
| GC_FEELINGTRACKER_ACCESS | Contains the text of the Google Cloud service account access JSON downloaded when the service account was created |
| FEELING_TRACKER_PROJECT_ID | The unique identifier of the Google Cloud project under which the Kubernetes Cluster and Container Registry were created. |
Using Jenkins to Deploy the Containerized NodeJS Web App to a Kubernetes Cluster on Google Cloud
Create a new project in Jenkins by selecting New Item from the Jenkins main page and then declaring the item as a Freestyle Project, as shown in Figure 16, below.
Set the Build Environment to Use Secret Information
Now we’ll set the Build Environment. Select Use secret text(s) or files(s) as shown below in Figure 19. Setting the build environment to use secret text(s) and files (s) allow Jenkins to create runtime environment variables that will be bound to the secrets we defined earlier.
Setting Access Information using the Jenkins Credential Plugin
We’ll set up the information that Jenkins will need to access Google Cloud by creating runtime environment variables that are bound to the secret credentials that created earlier using the Jenkins Credential Plugin. The two environment variables that we’ll create are:
- GC_ACCESS which binds to the secret credential GC_FEELINGTRACKER_ACCESS
- PROJECT_ID which binds to the secret credential FEELING_TRACKER_PROJECT_ID
The details of the binding process are described in the sections that follow.
Defining the User Access Credentials
To add the access information described in the JSON file associated with the Google Cloud service account created earlier, click the Add button in the Bindings section of the Jenkins build and select Secret text. Add the information as shown below in Figure 20.
Defining the Project ID
To add the Project ID associated with the Google Cloud service account created earlier, click the Add button in the Bindings section and add the information as shown below in Figure 21.
Build Timeout
Finally, we need to add information in the Bindings section that declares an overall timeout period for the build. Also, we need to tell Jenkins what to do upon timeout. Figure 22 below shows the details of the settings.
Defining the Build Steps
The way the Jenkins project builds the demo app into a container and then deploys that container to the Kubernetes Cluster is by adding the Execute Shell task as a build step in the project. Figure 23 below shows the build step for containerization and deployment with the Jenkins GUI.
printf %s "$GC_ACCESS" > "gcsecret.json"
gcloud auth activate-service-account --key-file=gcsecret.json
gcloud config set project ${PROJECT_ID}
docker build -t feelingtracker .
docker tag feelingtracker us.gcr.io/${PROJECT_ID}/feelingtracker:v1
gcloud --project=${PROJECT_ID} docker -- push us.gcr.io/${PROJECT_ID}/feelingtracker:v1
gcloud container clusters get-credentials feelingtrackercluster --zone=us-central1-a
kubectl run feelingtracker --image=us.gcr.io/${PROJECT_ID}/feelingtracker:v1 --port=3000 --replicas=3
kubectl get deployment
kubectl expose deployment feelingtracker --type=LoadBalancer
sleep 30s
kubectl get service | grep 'feelingtracker'Taking a look at the details of the Build Step process
The following describes the details behind each command in the build step process. The command below, which starts the process, takes the contents of the environment variable, $GC_ACCESS and saves it to a file, gcsecret.json. $GC_ACCESS is mapped to the credential secret that holds the contents of the JSON file that contains the service account information needed to access Google Cloud.
printf %s "$GC_ACCESS" > "gcsecret.json"
The next command uses the command gcloud auth from the Google Cloud SDK to authorize a command session on the Jenkins server with Google Cloud. The command uses the –key-file parameter to pass authentication information stored in the file, gcsecret.json to Google Cloud.
gcloud auth activate-service-account --key-file=gcsecret.json
The command gcloud config set project tells Google Cloud to set the scope of activity on Google Cloud to the project defined by the environment variable, PROJECT_ID.
gcloud config set project ${PROJECT_ID}
Now we’ll build the docker file using the docker build command. The command assumes that the present working directory has the Dockerfile with the build instructions.
docker build -t feelingtracker .
After building the container, we need to tag it with the exact path to the file system location in the Google Container Registry. Also, we’ll append a version definition, v1, to the end of the tag. This tagging step is required by Google Cloud. Notice that the environment variable, ${PROJECT_ID} is part of the file path definition.
docker tag feelingtracker us.gcr.io/${PROJECT_ID}/feelingtracker:v1
We push the container image to the Google Cloud Container Repository using the commands gcloud and docker.
gcloud --project=${PROJECT_ID} docker -- push us.gcr.io/${PROJECT_ID}/feelingtracker:v1
Now we download the access credentials for the Kubernetes cluster from Google Cloud to the Jenkins server. The command, gcloud container clusters get-credential automatically appends the cluster, user and context information relevant to the Kubernetes cluster feelingtrackercluster to the Kubernetes config file on the Jenkins server. Also, the command makes the context related to feelingtrackercluster the current context that kubectl will use.
Hint
What is a Kubernetes context? In Kubernetes, a context is a formal mechanism that binds a Kubernetes cluster to a Kubernetes user profile. The following example shows a context, minikube that binds the user, dicktracy@minikube to the cluster, minikube.
- context: cluster: minikube user: dicktracy@minikube name: dicktracy@minikube
Developers and admins using Kubernetes work in a cluster defined by a particular context and have operational permissions within that cluster that are defined by the user profile declared in the context. Go here for more information about Kubernetes contexts.
gcloud container clusters get-credentials feelingtrackercluster --zone=us-central1-a
We use kubectl run to have Kubernetes create a Kubernetes deployment using the demo application’s container image that has just been uploaded to the Google Container Registry.
kubectl run feelingtracker --image=us.gcr.io/${PROJECT_ID}/feelingtracker:v1 --port=3000 --replicas=3
The command, kubectl get deployment confirms that the deployment is running in the Kubernetes Cluster on Google Cloud. This is simply a convenience call.
kubectl get deployment
We execute kubectl expose deployment to tell Kubernetes to create a Kubernetes service that is bound to the Kubernetes deployment we just created.
kubectl expose deployment feelingtracker --type=LoadBalancer
It takes a while for Kubernetes to create a service and expose a public IP address that’s public on the internet. Therefore, we’ll wait 30 seconds for the exposure of the public IP address to be implemented. In some cases, a wait period of a few minutes might need to be set.
sleep 30s
The final build step runs kubectl get service filtering on ‘feelingtracker’ to visually determine that the service has been created and that a public IP address has been issued.
kubectl get service | grep 'feelingtracker'
Calling the Project that Runs the Ranorex Test
Once Jenkins containerizes the demo app and deploys it as service in the Kubernetes cluster, we’re ready to run the Ranorex tests against the demo app running as a website. We’re going to isolate the Ranorex testing into a separate Jenkins project that runs on a Windows slave server in Azure. This project is called FeelingTrackerRanorexTest. We’ll add a post-build step to call this separate build project when the current project finishes successfully. In order to add the post-build step, we go the section Post Build Actions at the bottom of the Jenkins project page. Click the Add post-build action button and then select Build other projects from the drop-down list that appears. (See Figure 24.) Enter the name of the project that’s going to be executed and select the option, Only if build is stable.
Next Steps
The next, final installment of this series describes how to configure the separate Jenkins project FeelingTrackerRanorexTest to test the demo application by executing the associated Ranorex from a Windows slave server running in the cloud on Azure. The build FeelingTrackerRanorexTest will run separately after the Jenkins project described in the article finishes deploying the demo application to Google Cloud.
