From Code to Production with GitOps

In this blog post we are going to introduce how we can get from code to production deployments with GitOps and how you can implement these patterns on OpenShift.

In the introductory blog post to GitOps we described some of the use cases that we can solve with GitOps on OpenShift. In today's blog post we are going to describe how we can leverage GitOps patterns to manage our applications life cycle on OpenShift.

We are going to explore the following use cases:

• Build and application from source using Tekton Pipelines and Git webhooks
• Deploy our application to different environments using Argo CD

NOTE: For this blog we are working with GitHub, but the same steps can be done with other Git servers, in case you decide to use a different provider, make sure you change the GitHub references on the demo files.

Environment

• One OpenShift 4.3 cluster
• ArgoCD used as the GitOps tool
• Tekton Pipelines used as the CI/CD tool
• Demo files:
  • Demo Repository
  • Tekton Files
  • App Config Files
  • App Config Stage Files
  • App Config Production Files

Adapting demo files to your inventory

1. Change user and password for Quay.io on quay-credentials.yaml
2. Change references to your image/repository on build-pipeline.yaml: 1, 2, 3 and 4
3. Change deployment image on these files: 1 and 2

Scenario

Our team is responsible of the development and deployment of a simple application that reverses words, we want that whenever a new commit hits the master branch a new Pipeline is executed, this Pipeline will take care of:

1. Download the new code
2. Run a code linter
3. Run the tests
4. Build the app on a container image
5. Push the container image to our container image repository

Our application is deployed on two environments, stage and production, in order to deploy the application we are following a GitOps model, in this case we have one repository with three branches:

• config Branch - Has the required files for deploying our application and that are common to all environments
• stage Branch - Has the required overlay files for deploying our application into the stage environment
• prod Branch - Has the required overlay files for deploying our application into the production environment

Whenever a new commit hits stage or prod branches, a webhook is sent to Argo CD and the application is synchronized by Argo CD.

GitOps Workflow

1. Developers push new changes to the application repository
2. The changes are tested and a new image is built automatically by a Tekton Pipeline
3. Developers request the deployment of the new image by sending a PR to the stage branch on code-to-prod-blog repository
4. The PR is reviewed and merged
5. The new image version is deployed automatically by Argo CD on stage
6. After testing the application on the stage environment a PR is sent to the prod branch to deploy the application on production environment
7. The PR is reviewed and merged
8. The new image version is deployed automatically by Argo CD on production

Pre-requisites

For this blog, we've used the following versions:

• OpenShift 4.2
• OpenShift Pipelines Operator v0.9.1
• Argo CD v1.3.6
• Quay.io Account

The deployment of this products/projects is outside of the scope of this blog post.

Forking the demo repositories

1. Go to https://github.com/mvazquezc/code-to-prod-blog and make a fork of the repository.
2. Go to https://github.com/mvazquezc/reverse-words and make a fork of the repository.

You will be using your forks for the demo.

Configuring Tekton Build Pipeline

As we said earlier in the post, we have a Pipeline that will take care of build new container image versions of our application whenever a code change is detected.

We will create the required Tekton objects:

NOTE: You need to provide your own credentials in quay-credentials.yaml file. And modify the PipelineResources in build-pipeline.yaml.

# Clone your repository fork
git clone <git_url_to_your_fork>
# Access your fork
cd code-to-prod-blog
# Checkout CI branch (contains tekton files)
git checkout ci
# Create a namespace for storing our configurations
oc create namespace tekton-reversewords
# We need the credentials for our container image registry (remember to add the your credentials to this file)
oc -n tekton-reversewords apply -f quay-credentials.yaml
# We need a ServiceAccount with access to the credentials created before
oc -n tekton-reversewords apply -f pipeline-sa.yaml
# Linter task - It will run a linter against our code
oc -n tekton-reversewords apply -f lint-task.yaml
# Test task - It will run the tests defined in our repository
oc -n tekton-reversewords apply -f test-task.yaml
# Build task - It will create and push the new container image to the container registry
oc -n tekton-reversewords apply -f build-task.yaml
# Pipeline - It defines the Build Pipeline that will execute previous defined tasks in an specific order with specific parameters (remember to point to your own repos)
oc -n tekton-reversewords apply -f build-pipeline.yaml
# Webhook - Will receive webhooks from GitHub and run the pipeline with the parameters received on the webhook
oc -n tekton-reversewords apply -f webhook.yaml

Configuring Argo CD Applications

Argo CD will take care of deploying new changes onto our cluster, we have three branches within the code-to-prod-blog repository with the required files for Argo CD to deploy our application on each environment:

• Branch config - Base config files
• Branch stage - Stage environment override files
• Branch prod - Production environment override files

Below the required steps to configure Argo CD applications:

NOTE: We will be using the argocd cli, these steps can be done from the WebUI as well. The examples below use my repository, you're supossed to create your own using the demo files described on the environment section.

1. Define git repository

   argocd repo add https://github.com/mvazquezc/code-to-prod-blog.git

2. Define stage application

   argocd app create --project default --name reverse-words-stage \
   --repo https://github.com/mvazquezc/code-to-prod-blog.git \
   --path . \
   --dest-server https://kubernetes.default.svc \
   --dest-namespace reverse-words-stage --revision stage \
   --sync-policy automated

3. Define production application

   argocd app create --project default --name reverse-words-production \
   --repo https://github.com/mvazquezc/code-to-prod-blog.git \
   --path . \
   --dest-server https://kubernetes.default.svc \
   --dest-namespace reverse-words-production --revision prod \
   --sync-policy automated

Now we have ArgoCD configured and ready to deploy our apps, now it's time to configure the required webhooks.

Configuring Git WebHooks

We want to build our application when new commits hit the master branch on our code repository and deploy it when new commits hit the config/stage/prod branch on our config repositories.

Configuring webhook for Build trigger

1. We need to get the url where we will send the webhook

   # Get the route
   oc -n tekton-reversewords get route el-webhook -o jsonpath='{.spec.host}'

2. Go to the Git repository for the reverse-words application and configure the webhook

   1. GitHub Repository -> Settings -> Webhooks

   2. Configure it using the route from step 1:

      

Configuring webhook for Stage deployment

1. The url for the webhook in this case will be the argocd server route/external url + "/api/webhook", e,g; https://argocd.example.com/api/webhook
2. Go to the stage deployment repository and configure the webhook

   1. GitHub Repository -> Settings -> Webhooks

   2. Configure it using the route from step 1

      

Configuring webhook for Production deployment

1. The url for the webhook in this case will be the argocd server route/external url + "/api/webhook", e,g; https://argocd.example.com/api/webhook
2. Go to the production deployment repository and configure the webhook

   1. GitHub Repository -> Settings -> Webhooks

   2. Configure it using the route from step 1

      

You can also use the below script to define the webhooks

export GIT_WEBHOOK_URL=$(oc get route el-cicd -o jsonpath='{.spec.host}' -n $NAMESPACE)
echo "https://$GIT_WEBHOOK_URL"

export ARGOCD_WEBHOOK=$(oc get route argocd-server -n argocd -o jsonpath='{.spec.host}')/api/webhook


#Set the GIT_REPO_NAME to name of the Code Git repo like reverse-words
export GIT_REPO_NAME='code-repo'
export GIT_USERNAME='ch-stark'
export GIT_TOKEN= YOURTOKEN 

#Run curl to create the webhook for the build trigger
curl -v -X POST -u $GIT_USERNAME:$GIT_TOKEN \
-d "{\"name\": \"web\",\"active\": true,\"events\": [\"push\"],\"config\": {\"url\": \"https://$GIT_WEBHOOK_URL\",\"content_type\": \"json\",\"insecure_ssl\": \"0\"}}" \
-L https://api.github.com/repos/$GIT_USERNAME/$GIT_REPO_NAME/hooks



export GIT_REPO_NAME='prod-repo/stage-repo'


curl -v -X POST -u $GIT_USERNAME:$GIT_TOKEN \
-d "{\"name\": \"web\",\"active\": true,\"events\": [\"push\"],\"config\": {\"url\": \"https://$ARGOCD_WEBHOOK\",\"content_type\": \"json\",\"insecure_ssl\": \"0\"}}" \
-L https://api.github.com/repos/$GIT_USERNAME/$GIT_REPO_NAME/hooks

Testing our Workflow

Now we have our two Argo CD apps created, the Tekton build pipeline defined and the webhooks configured.

1. Let's send a change to our application code and see what happens:

   

2. The build pipeline generated a new tag for our image:

   

3. Let's update the deployment on stage branch to use the image that was just created so Argo CD updates the deployment:

   

4. Now we have the image running on stage but not in production:

   

5. After updating production branch we will get the same version on bot environments: