Standard Operating Environment – Part II: Workflows in Detail

This article was originally posted on OpenSourceArchitects.org.

Introduction

In the previous article in this series we introduced the basic concepts of a Standard Operating Environment (SOE). Here we go into further details of the workflows and processes involved in an SOE. In the third part of this series we will discuss the implementation and operation of an SOE in practice.

Build Lifecycle

Previously we discussed how a build goes through the following lifecycle stages:

• Inception
• Development
• Release
• Maintenance
• Retirement

We will model each of these lifecycle stages as an Activity. As discussed in the previous article is usual to have multiple builds within the organization, each build potentially at a different lifecycle stage. NB Comments in these activity diagrams refer to how activities might be implemented in Red Hat Satellite.

Inception Activity

The inception activity can be modeled as shown below:

As discussed in the previous article, every build is identified by a unique name and settling the name is the first step of Inception. The name will be static throughout the entire lifecycle of the build and used to reference it in all documentation, communications etc.

The next step is to create the repositories that will be used to contain the build. Typically, these consists of a software repository to contain (in the Red Hat world) RPMs, an SCM repository (such as one or more git repositories (or branches on existing repositories) to contain text-based artifacts such as deployment templates (kickstart files, snippets, cloud-init scripts) and configuration information (Puppet modules or Ansible playbooks and roles). Finally, a requirements tracker, such as a JIRA project, is created to manage the build development process.

The two final stages of the activity are to create test machines and a development build plan. A common scenario is to create one test machine per profile and many organization will also create hardware (as opposed to virtual) test machines in order to increase test coverage to things like firmware and hardware compatibility. The build plan to be created will be used to automate integration and testing of the build in a CI engine such as Jenkins or Bamboo.

Once the Inception of the build is complete and the build development framework is in place, the Development Activity is triggered.

Development Activity

The development activity can be modeled as shown below:

The Development Activity starts with requirements gathering and task backlog creation. The need for a a structured requirements gathering process is clear when the expectations placed on a modern SOE build are noted. The build will be required to:

• Run on multiple platforms: hardware, virtualized, and cloud.
• Support a large variety of 3rd party software, and be aligned with the support matrix of each vendor
• Integrate with the organization’s authentication, security, monitoring, logging, backup, storage, and network management systems
• Meet non-functional requirements, such as performance and reliability requirements, for critical workloads

The complex requirements that are required of a modern SOE merit a more formal approach to requirements gathering and tracking, and of course many of the more static requirements will simply be inherited from earlier builds so implementation in the new build is likely to consist purely of ensuring successful test execution.

The main loop of the Development Activity consists of writing tests, software and configurations, integrating them, and testing the results; in other words a typical development-test loop. Many organizations will choose to use an Agile methodology, often Scrumban, to manage this process. A CI build plan implemented in Jenkins, Bamboo, or equivalent handles the creations of any artefacts that need to be built (such as RPMs or Puppet Modules), the integration of upstream content (such as Red Hat Enterprise Linux errata), the presentation of the new build version to test servers (via Red Hat Satellite Content Views), the re-deployment of the test servers and the execution of automated tests on the test servers.

Once all requirements have been implemented, and all tests are passing, the Release Activity is triggered.

Release Activity

The Release Activity can be modeled as shown below:

This is a simple linear activity that production-ises the build. Along with the normal documentation tasks that might be expected such as publishing release notes and a defined support lifecycle for the build – including the all-important EOL date – the build is exposed to server environments. In the case of the Red Hat Satellite, this would be achieved by promoting the build content view into a lifecycle environment in which the targeted servers reside. Finally, servers that have been scheduled to be migrated to this build are re-assigned to it (Content View reassignment in the Red Hat Satellite world) and updated. It is important to note that SOE does not mandate the updating of all servers to the latests build – as described in the previous article it is expected that multiple builds would be supported at any one time.

Maintenance Activity

The Maintenance Activity can be modeled as shown below:

This activity is similar to the Development Activity but concerns itself with the ongoing maintenance and updating of a build that is in production. There are in fact two activities under consideration:

• The triage and integration of updated vendor content e.g. Red Hat Enterprise Linux errata
• The resolution of issues raised by users

A general triage plan has been shown, however many organizations will use a risk plan to determine how frequently, or if it all, vendor patches are incorporated into the build, for example (using the security errata ratings used by Red Hat):

• Critical security errata are incorporated into the build within 48 hours
• Serious security errata are incorporated into the build within 7 days
• Moderate and Low security errata are incorporated into the build within 30 days
• Bugfixes are only incorporated into the build if there is an operational reason to do so i.e. they address a bug which is causing loss of service, otherwise they are deferred until the next build
• Enhancements are always deferred until the next build

Likewise the resolution of issues fixed by users is staged for incorporation into the current build, or postponed until the next build – for example a fix for a minor problem that does not cause operational loss may not warrant the release of a new build version.

It is important to note that every time the Maintenance Activity loop completes this results in the release of a new Build Version, not of a new build. The unique build name is unchanged, it is only the version of the build that increments. Red Hat Satellite supports this with Content View versioning.

Once a new Build Version has successfully tested it is exposed (for example via Content View promotion) to client servers which can then be updated as required.

Retirement Activity

The Retirement Activity can be modeled as shown below:

This is a simple linear activity to take the build out of production. As the Maintenance Activity for this particular build will be terminated, it is important that all servers currently registered to the build are migrated to a more current build.

Connect with Red Hat Services

Learn more about Red Hat Consulting
Learn more about Red Hat Training
Learn more about Red Hat Certification
Subscribe to the Training Newsletter
Follow Red Hat Services on Twitter
Follow Red Hat Open Innovation Labs on Twitter
Like Red Hat Services on Facebook
Watch Red Hat Training videos on YouTube
Follow Red Hat Certified Professionals on LinkedIn