An architect's guide to DevOps pipelines: Continuous integration & continuous delivery (CI/CD)

Constructing a DevOps pipeline is an essential part of a software architect's process when working in a software engineering team. In the past, as I participated as a technical interviewer at Red Hat, I was quite surprised to find very few people could clearly describe a DevOps pipeline and a continuous integration and continuous deployment (CI/CD) pipeline. You will understand what these pipelines are by reading this article.

If you prefer video, check out my summary of this information. 

What is DevOps?

Let's have a little discussion about what DevOps is before jumping into the DevOps pipeline. In common practice, DevOps is a software development methodology often associated with specific practices and tools that help implement those practices. It is mainly focused on increasing the frequency of software deliveries to build more resilient systems.

You can think of DevOps like other software development processes such as agile, Lean, scrum, or waterfall.

[ Download the Enterprise automation in a DevOps world checklist. ]

One of the most approachable books written about the DevOps process is The Phoenix Project by Gene Kim. A reference page in the back of the book explains the software deployment frequencies of different companies. It states that a typical enterprise releases software only once every nine months. In contrast, Netflix, releases software updates thousands of times per day due to its well-constructed DevOps process.

More releases without making significant changes to team processes can lead to less resiliency rather than more. DevOps processes that focus only on frequency of delivery can be misleading and undesirable. A DevOps team also needs to emphasize the system's reliability, safety, observability, and resiliency. These aspects are positively affected by more frequent releases, as discussed in Accelerate by Nicole Forsgren, Jez Humble, and Gene Kim.

[ Check out 8 books to boost your enterprise architecture career. ]

Now that you understand what the DevOps process is, what does DevOps promise to deliver? To understand this, you need to understand what the DevOps pipeline is.

What is a DevOps pipeline?

A DevOps pipeline is the bread and butter of the DevOps process. The term is used most often to discuss the tools, processes, and automation frameworks used to build software artifacts. The most well-known DevOps pipeline, and the one that kicked off the DevOps trend, is Jenkins, which is an open source tool built in the Java programming language. Now, we have many DevOps pipeline tools like Argo, Harness, GitHub Actions, and Travis CI.

A DevOps pipeline is important to understand based on the organization that will use it. These teams can be largely broken down into two aspects: infrastructure (operations) and code development (development). Which role you belong to depends on a number of different factors, including the existing DevOps pipeline, the size of your company, your position at the company, but it is most connected to whether you produce software (development) or manage where software is deployed (operations).

For example, imagine that your company is a startup that only has a few engineering team members. The company has no previous DevOps pipeline. As an architect, your responsibility is to create an elegant solution that improves every software delivery aspect. You need to come up with at least the minimum viable product (MVP) pipeline that delivers the infrastructure aspect as well as the code deployment aspect.

If you work in a large enterprise with an existing software delivery methodology, you will need to focus on both the infrastructure side as an operations engineer and the code delivery side as an application developer. That need to span this gap is the true goal of DevOps. Understanding what stage your team's DevOps process is at, your role, and the size of your team helps you determine what you need to focus on when you join an engineering team.

Now we finally get to the exciting part: Learning how to design a DevOps pipeline. First, I will cover the infrastructure side and then move on to the code development side.

Create a DevOps pipeline for infrastructure

As an infrastructure engineer—also called an operations engineer (Ops) or a DevOps engineer—you are responsible for building the environment necessary to host and run applications. Most of the time, this means a cloud environment, which can be either running on top of a virtual machine (VM) or in containers.

[ Compare containers vs. virtual machines]  

The architecture diagram below illustrates a general DevOps pipeline process applicable to an infrastructure engineer or an architect.

The high-level, step-by-step process is:

1. A DevOps pipeline job is triggered.
2. The pipeline job executes a task to check out the code from a source control management (SCM) tool, like GitHub or GitLab. This SCM includes the DevOps pipeline script and configuration management tools (for example, Ansible or Bash) to be executed by the DevOps pipeline job steps.
3. The DevOps pipeline authenticates to an SCM and checks out the code.
4. The DevOps pipeline executes the job based on the steps outlined by the DevOps pipeline.
5. To build a cluster or perform any infrastructure-related activity, an ephemeral pipeline agent is usually needed. A DevOps pipeline authenticates with an image registry (such as Quay, Artifactory, DockerHub) and pulls an image that is used for spinning out the pipeline agent.
6. A temporary pipeline agent, which usually already has the scripts, libraries, and tools (another use for Ansible) necessary to execute the scripts from the SCM, spins out and performs actions based on the DevOps pipeline script's instructions.
7. If the requested infrastructure pipeline action is about building or customizing an image, the temporary pipeline agent may pull the additional images from the image registry or push them to the image registry. The pipeline agent shuts down after the execution.
8. If the requested infrastructure pipeline action is about creating, updating, or deleting a cluster, the temporary pipeline agent may authenticate and then perform the infrastructure operation activity. The pipeline agent shuts down after the execution.
9. After a pipeline action completes, the pipeline notifies the engineering team or stakeholders of the result, either through an email or another messaging platform like Element, Microsoft Teams, or Slack.

Infrastructure DevOps pipeline technology stacks

Let's explore some commonly used technology stacks in the infrastructure DevOps pipeline.

First, here are a few examples of the DevOps pipeline tools. Note that these tools are usually equivalent to the DevOps pipeline for the application developers:

• Jenkins
• Harness
• Argo
• GitLab
• GitHub Action

You also have several choices for SCM tools. Here are a few examples:

• GitHub
• BitBucket
• GitLab
• Subversion

If you want to store operating system images or container images, you need some type of image registry. Here are a few options you can explore:

• Quay
• Artifactory
• Nexus
• DockerHub

Most importantly, to perform the infrastructure-related activities, such as provisioning a cluster or building a custom VM image, you need to have configuration management tools and shell scripts, such as the ones below:

• Ansible: This configuration management tool is excellent at installing packages, creating a VM image, and carrying out other VM-related operations.
• Bash script: While it takes more work to make it idempotent, Bash can be useful for all types of operations tasks.

Lastly, you need a cloud provider or an environment to deploy and manage your virtual machine, a Kubernetes container, or a cluster. Any public cloud provider will work, but you can start with a virtual machine or a container environment running on your local computer.

[ Learn how to build a flexible foundation for your organization. Download An architect's guide to multicloud infrastructure. ]

Most common infrastructure DevOps activities

Although it is impossible to list all infrastructure activities that can be performed by the DevOps pipeline, here is a list of some of the most commonly used infrastructure tasks and tools that are good for the task:

• Provisioning a cluster (oc or kubectl)
• Deleting a cluster (oc or kubectl)
• Making a change in a cluster (oc or kubectl)
• Building a container image (docker or podman)
• Building a VM image (Ansible) 
• Installing packages and performing post-installation activities (Ansible)
• Making a change to a network in a cluster (Cluster Network Operator)
• Provisioning storage in a cluster (oc for dynamic provisioning)
• Backing up the state of a running environment (Operators like this one)
• Restoring a previously-working state of a running environment (possible on OpenShift)

DevOps pipeline for application developers

Let's switch gears slightly to explain how the DevOps pipeline for application developers works. If the DevOps pipeline for the infrastructure team is centered around how to provide a stable environment, the DevOps pipeline for the application developers is designed to deliver software updates as often as possible. (Remember from earlier: more frequent update leads to greater stability according to research.)

Here is a possible architecture diagram illustrating the DevOps pipeline for the application developers. Note that this is a simplified diagram in a non-container context. With the container-based approach, additional steps can be introduced.

The high-level step-by-step process is listed below:

1. A DevOps pipeline job is triggered.
2. The pipeline job executes a task to check out the code from a Source Control Management (SCM), like GitHub, BitBucket, or GitLab. This SCM includes the DevOps pipeline script and the source files for a specific programming runtime (e.g., Java, Python, Go, etc.). If the software building process involves deployment to a container/Kubernetes environment, the SCM repo may include the Dockerfile or Source-to-Image (S2i) configuration file.
3. The codes get checked out, and the DevOps pipeline script and the necessary source codes are now inside the DevOps pipeline's temporary workspace.
4. The pipeline executes the step-by-step tasks based on the DevOps pipeline scripts. Depending on the programming runtime, this usually invokes build scripts using tooling such as Maven, Grunt, or Makefile.
5. In addition to the unit test, the pipeline runs additional end-to-end tests and other quality confirmations, and then publishes the result through a tool like SonarQube. If the build fails or succeeds, the pipeline should also notify the developer team about the result.
6. If the application is successfully built and passes all the tests, the packaged application can be deployed to a virtual machine or a container in a Kubernetes environment.

Application developer DevOps pipeline technology stacks

Many tools used by the application developer-focused pipeline are similar to those of the infrastructure-focused pipeline. One addition is a code quality tool. A few examples of the code quality tools are shown below:

• SonarQube
• Semmle
• Lint
• Checkstyle

To build software, you need a build tool. The recommended tool differs based on the programming runtime. Here are some common build tools for popular programming runtimes:

• Java: Apache Maven, Apache Ant, Apache Gradle
• JavaScript:  Grunt, Webpack, Gulp
• Python: Fabricate, Pynt, Fabric

Most common application development DevOps activities

Here are a few common application development DevOps tasks that can be performed by the pipeline:

• Compile and build software source code.
• Run the tests and generate test results.
• Deploy the code package to a virtual machine or a container.
• Merge the code back to an SCM branch.
• Close a User Story or a task in an Agile tool.

What is the CI/CD pipeline?

The idea of separating the CI process from the CD process has not been a common practice adopted by the software industry until recently. The most well-known DevOps pipeline tool, Jenkins, started when the distinction between CI and CD was vague. As the software engineering process evolved, the benefits of separating the CI process from the CD process have become apparent.

• Continuous Integration: Developers merge their changes back to the main branch as often as possible. The developer's changes are validated by creating a build and running automated tests against the build.
• Continuous Delivery: Releases new changes to one or more environments, including production. This means that on top of having automated testing, you also have an automated release process, and you can deploy your application at any point in time by clicking on a button.
• Continuous Development: Every change that passes all stages of your production pipeline is released to production. There's no human intervention, and only a failed test will prevent a new change to be deployed to production.

While a team can use the Jenkins pipeline to implement the entire CI/CD process, the industry trend now leans toward adopting a separate DevOps pipeline tool for the CD process. For example, while one company may still use Jenkins to build a software or a VM image, they may use Harness or Argo to deliver the package to a cluster or VM environment.

Putting the pipeline together

In this article, you learned about the heart of DevOps and DevOps pipelines. You also learned how to create DevOps pipelines from two different perspectives: One with infrastructure engineers in mind and one for the application developers. You also saw the difference between continuous integration (CI) and continuous delivery (CD) processes and how the industry trend is to separate the two.

When applied to both operations and application development, these technologies lead to more deployments per day, which has been shown by research to lead to more resilient systems. That's why DevOps is powerful.