Ansible Automation Platform for OpenShift Virtualization in Multi-cluster Environment

The rise of Red Hat OpenShift has led organizations to adopt a centralized hub cluster model for automating their entire IT operations. This hub cluster, typically placed in a management segment, hosts centralized components like Red Hat Ansible Automation Platform, Red Hat Advanced Cluster Management for Kubernetes, and Red Hat Advanced Cluster Security for Kubernetes.

The increasing adoption of Red Hat OpenShift Virtualization presents a need for automated solutions for virtual machine (VM) Day-1 and Day-2 activities. A centralized Ansible Automation Platform model within the hub cluster empowers automation across various OpenShift clusters spanning multiple zones, data centers, and clouds.

This approach focuses on automating VM creation on remote OpenShift clusters by leveraging a centrally deployed Ansible Automation Platform on the hub cluster. Virtualized workloads are expected to run on separate remote clusters.

In this article, Ansible Automation Platform will be configured to communicate with a remote OpenShift cluster to manage virtual machines. This communication will enable Ansible Automation Platform to:

• Create a virtual machine on remote cluster
• Perform post-provisioning tasks like installing necessary packages and starting services via container groups
• Set up network resources for reachability

The setup is as follows:

• OpenShift Hub Cluster
• Ansible Automation Platform deployed on the OpenShift Hub Cluster
• OpenShift Remote Cluster with OpenShift Virtualization

Diagram illustrating the environment components

Before we dive into the specifics of Ansible Automation Platform, let's unpack a few essential concepts. Grasping these fundamentals will be key to understanding how Ansible Automation Platform works and maximizing its benefits.

Container Groups: represent a reference to an OpenShift/K8s cluster where Ansible Automation Platform can launch pods in a predetermined namespace to carry out the execution of a job template/playbook within the remote Red Hat OpenShift environment. The pod only exists temporarily during the execution phase and enables a clean environment for every job run.

Kubevirt Inventory: Ansible inventory plugin for discovering virtual machines running on Kubernetes/OpenShift via KubeVirt or OpenShift Virtualization

Leveraging Certified Content for Red Hat OpenShift Automation

This workflow utilizes certified collections from Ansible automation hub to streamline interactions with Red Hat OpenShift and OpenShift Virtualization.

• red hat.openshift: Provides modules for creating Kubernetes resources on your OpenShift cluster.
• redhat.openshift_virtualization: Facilitates VM creation on OpenShift Virtualization and includes an inventory plugin for synchronizing OpenShift VMs with Ansible hosts.
• kubernetes.core: Provides modules for retrieving information regarding OpenShift/Kubernetes resources

Ansible Automation Hub: Your Central Hub for Content

Ansible automation hub is a centralized repository for discovering, downloading, and managing pre-built, supported Ansible Content Collections. These collections bundle modules, plugins, roles, and documentation to automate various IT tasks.

Configuring Ansible Automation Platform for Content Access

To utilize certified collections within the automation controller, follow these steps:

Obtain Hub Credentials: Login to Ansible automation hub at https://console.redhat.com/ansible/automation-hub. Navigate to "Automation Hub" -> "Connect to Hub" and click "Load Token" under "Offline Token" to generate a token. Copy down the token, server URL, and SSO URL. 

Retrieving Automation Hub token, server URL, and SSO URL

Add Credentials: In the Ansible Automation Platform web console, navigate to "Resources" -> "Credentials" and click "Add." Select “Ansible Galaxy/Automation Hub API Token” as the “Credentials Type”. Enter the previously obtained token, server URL, and SSO URL.

Adding the Ansible automation hub credentials in Ansible Automation Platform

Update Organization Settings: Navigate to "Access" -> "Organizations," select your organization, click "Edit," and go to "Galaxy Credentials." Click the "Search" icon, select the newly created Ansible automation hub credential, and move it to the top of the list to prioritize its use.

Updating the organization settings to add the Ansible automation hub credentials

Now Ansible Automation Platform is configured to leverage Red Hat Ansible Certified Content Collections from Ansible automation hub and we can move on to creating the workflow resources.

Create the Ansible Automation Platform Project

A project referencing our Git repository hosting the necessary playbooks will be created on Ansible Automation Platform so that we can utilize these playbooks in our dynamic kubeVirt inventory as well as the Ansible job templates that we will create. The project has been set up using this public GitHub repository. The requirements.yml file included in the repository are referencing the necessary collections that will be retrieved from Ansible automation hub.

Adding the RHAPP project

Create secret of SSH key

Ansible Automation Platform will require SSH access to the VMs running on the target cluster. To do so, a predefined SSH private key will be added as “Credentials” in Ansible Automation Platform to SSH to the target VM. The public key will be injected at VM creation time via cloud-init so that Ansible Automation Platform can access it.

Adding the predefined SSH key to used to connect to the VMs

Create secret for the remote OpenShift cluster

Since the job template will create a VM on the target OpenShift cluster, it needs to authenticate against it. We will add the target OpenShift cluster’s API endpoint and a token as “Credentials” in  Ansible Automation Platform. This should be a token associated with a user who has privileges to create the necessary resources such as VMs, services and routes. To obtain Red Hat OpenShift credentials, log in to the OpenShift web console and navigate to the top right section. Click on your username to access the dropdown menu and select 'Copy Login Command'. After authenticating, the OAuth token and API server URL can be copied.

Copy Login Command’ from the OpenShift console

Retrieving the OAuth token and the API server URL

Adding the credentials for the remote OpenShift cluster

Create Container Group Service Account (SA) on Target Cluster

Ultimately we need the SA token, and a namespace on the target cluster to launch the automation job pods in where the SA has the necessary permissions. This will be added as a container group in the Ansible Automation Platform configuration. The Ansible documentation provides detailed information on how to create container groups and the necessary prerequisites. In our example, we have created a project called “remote-aap” on the remote OpenShift cluster where the SA has the necessary permissions.

Create the Container Group on Ansible Automation Platform

A container group is created on Ansible Automation Platform utilizing the previously created credentials which reference our remote OpenShift cluster and in the pod spec override, the namespace has been updated to the “remote-app” project that was created.

Creating the container group on Ansible Automation Platform

Create Custom Credentials for KubeVirt Inventory

In order for Ansible Automation Platform to discover OpenShift Virtualization VMs as inventory, we will leverage the KubeVirt inventory plugin. The KubeVirt inventory plugin requires credentials in order to authenticate to the target cluster and retrieve the VMs to import them as “Inventory” in Ansible Automation Platform. Ideally, we should be able to use the builtin OpenShift credentials we created in the previous step, however due to the current implementation of the KubeVirt inventory module, it requires the authentication credentials to be available through environment variables. This will be addressed in Ansible Automation Platform version 2.5 and the KubeVirt inventory will be a built-in inventory source with no additional required configuration.

The custom credentials type created capture the OpenShift API endpoint and the bearer token and then inject them as environment variables “K8S_AUTH_HOST”, “K8S_AUTH_API_KEY”. These will be used by the KubeVirt inventory plugin to authenticate against the remote OpenShift cluster.

Creating the custom credentials type needed for the KubeVirt inventory plugin

Creating the credentials based on the newly created credential type

Now we have all the required credentials, SSH keys and tokens in place, we can proceed to create the automation resources on Ansible Automation Platform.

Create the KubeVirt Inventory & Inventory Source

The KubeVirt inventory will be responsible for retrieving VMs running on the target OpenShift cluster and importing them as “Inventory” so  Ansible Automation Platform can connect and configure them. The inventory source has been set as “Sourced from a Project” and in the Inventory file we are referencing the KubeVirt inventory from our git repo.

Creating the inventory

Creating the inventory source

Create Job Template for VM creation

In this playbook we are creating a project “workloads” to host our new VM, creating the VM and  injecting the SSH key via cloud-init and finally waiting until the VM is up and running and an IP address has been assigned to it.

“Create VM” job template

Sync Inventory

This step is necessary so that the newly created VM is visible and imported under the Ansible Automation Platform inventory. Here is where the KubeVirt inventory plugin is leveraged to connect to the remote OpenShift cluster and import the VM as inventory in Ansible Automation Platform.

Create Job Template for Day-2 Activities

In this playbook, Ansible Automation Platform will connect to the newly created VM and install the httpd package to start the service. In the playbook, we are limiting execution by setting the “hosts: namespace-workloads” to target only the newly created VM in the “workloads” namespace. Once inventory is synced, the KubeVirt plugin will created different groups that can be used to easily identity and target specific hosts

Discovered inventory groups using the KubeVirt plugin

Create job template Networking Resources

In this playbook, we are creating a service and route to expose the httpd web server running on the VM to the outside world so that external clients can connect to it.

“Create Network Resources” job template

Create the Ansible Automation Platform Workflow

After creating all the necessary job templates., we can build a workflow to connect the job templates and have them execute in a sequential manner to achieve the required automation use cases.

With all the pieces in place, we are ready to launch the workflow!

Launching the “Create Webserver OCP Virt” workflow

The first job template handles creation of the project and the VM and waits for the IP to be assigned. Next up, the inventory sync job kicks in to retrieve the newly created VM and import it into the Ansible Automation Platform inventory. Afterwards, Ansible Automation Platform will launch a pod on the remote cluster to execute the post provisioning job template. This is executed via the “container groups'' concept where we can specify remote OpenShift clusters for job execution. Since our VM is using the default OpenShift networking, SSH access to the VM is only available within the OpenShift cluster by default.  Ansible Automation Platform, which is residing on the management OpenShift cluster, cannot directly connect to it. By delegating this job to a container group in the target OpenShift cluster, Ansible Automation Platform will communicate to the target cluster via the OpenShift API to launch a pod to execute the necessary automation instructed in the playbook. In turn, this job pod will SSH to the target VM as defined in the playbook/job template in order to install the packages and start the services.

If we are quick enough, we can view the pod details on the remote cluster in the remote namespace that is configured in Ansible Automation Platform to launch the container group in.

Container group pod during execution on remote cluster

Finally, we can retrieve the newly created route URL from the remote OpenShift cluster and validate the process by connecting to the web server residing on the VM via the browser.

Route created by Ansible Automation Platform on remote cluster pointing to the newly created VMs web server

Connecting to web server running on the newly created VM

This article showcased the power of Ansible Automation Platform in a multi-cluster environment, automating VM Day-1 and Day-2 activities. We've provided a foundational example: creating a single VM on a remote cluster with a basic httpd web server. However, the true potential lies in its scalability. This model can be extended to orchestrate the deployment of complex, multi-VM applications with intricate components across multiple OpenShift clusters.

Resources:

• https://console.redhat.com/ansible/automation-hub/repo/published/redhat/openshift_virtualization/content/inventory/kubevirt/

• https://console.redhat.com/ansible/automation-hub/repo/published/redhat/openshift/content/module/k8s/

• https://console.redhat.com/ansible/automation-hub/repo/published/kubernetes/core/content/module/k8s_info/

• https://kubevirt.io/2023/Managing-KubeVirt-VMs-with-Ansible.html

• https://github.com/ahmbas/ansible-ocp-virt