Migrating from a virtualization platform to a hybrid cloud solution: A hands-on account

Many companies and organizations are transitioning from traditional virtualization platforms to hybrid cloud solutions. Recently a group of Red Hat engineers took this journey, performing a large-scale migration from Red Hat Virtualization to Red Hat OpenShift, using Migration Toolkit for Virtualization, an advanced tool for migrating virtual workloads to OpenShift.

Our Red Hat Virtualization environment successfully ran hundreds of virtual machines (VMs) for more than a decade—some used by engineers for development and testing purposes, and others for production workloads. However, with Red Hat Virtualization approaching its end of life, we decided it was time to migrate our VMs to OpenShift.

Considering the nature of the VMs that were running in Red Hat Virtualization and our desire to minimize disruption to end users, we chose to continue running the workloads within VMs on OpenShift using OpenShift Virtualization. To achieve that, the migration was performed with Migration Toolkit for Virtualization 2.4, which was the most advanced version at the time with enhanced migration features.

Our journey from Red Hat Virtualization to OpenShift exposed us to challenges, including our own and those experienced by users. The rest of this article talks about how we addressed them and offers other insights from our experience.

Preparation and planning

Thorough planning was essential for a successful migration. As we began planning the migration, we realized we had several challenges.

First, we had to make sure the target OpenShift cluster had enough resources to cope with the migrated workloads. We started by identifying VMs that were not recently used in Red Hat Virtualization and those whose owners asked they not be migrated to the new OpenShift environment. These VMs were not included in the migration. We then made sure we had enough storage space in OpenShift for the disks of the VMs that we planned to migrate. We also made sure there were enough IP addresses to allocate for the VMs we planned to migrate in the target VLAN in OpenShift.

Second, we realized that we needed to bridge between the provisioning model we used in Red Hat Virtualization and that in the OpenShift Virtualization environment. In Red Hat Virtualization, new VMs were assigned for users by administrators. In contrast, in OpenShift Virtualization users are assigned to projects by administrators and can create VMs in their projects themselves.

Third, in Migration Toolkit for Virtualization, both the source and target providers are set with user credentials. This way, users define a source provider from which to migrate VMs they have permissions for, and a target provider on which to create VMs on their behalf. In our case we preferred to migrate all the VMs at once rather than ask each user to migrate their own. Therefore, we had to operate with users that had administrator privileges on both platforms, allowing them to access all the VMs in Red Hat Virtualization and to create VMs in all projects in OpenShift.

We found out that fetching which VMs were recently used, in our case over the past year, and the size of their disks to determine whether the target OpenShift cluster had sufficient storage capacity, was not a trivial task. In addition, getting the owners of the VMs from Red Hat Virtualization to create corresponding projects in OpenShift in order to bridge the different provisioning model required additional effort. We addressed these challenges by implementing Python scripts.

The first group of scripts gathered data from Red Hat Virtualization using oVirt SDK. From this data, we identified when VMs were last used and filtered out the VMs that were not used in the past year. Once we had the final list of VMs that would be migrated, we gathered details about the size of their disks and their owners. The data we got from Red Hat Virtualization showed us that we didn't have enough storage space in the OpenShift cluster for storing the migrated VMs' data. To address this, we increased the storage on the OpenShift cluster by adding a new storage class.

A second group of scripts was used to prepare the projects on the OpenShift cluster using the Kubernetes API. Using the data we got from the first set of scripts, we identified the set of owners of each of the migrated VMs. For each set of owners, we then created a project in the OpenShift cluster and assigned it to these owners by granting them the ClusterRole with admin permissions.

To illustrate this, let’s take a VM called shared_vm that is owned by two Red Hat Virtualization users, Alice and Bob. In OpenShift, we create a project named alice-bob-ns, for which both Alice and Bob have permissions. We then migrate the VM, shared_vm, to the alice_bob_ns project. Other VMs that are owned by Alice and Bob in Red Hat Virtualization can also be migrated to the same project, alice_bob_ns. Alice and Bob can be assigned to other projects with VMs from Red Hat Virtualization that they own solely or with other users. This approach allows us to address the different provisioning models in the Red Hat Virtualization and OpenShift environments.

With this problem solved, we moved on to migrating the virtual machines.

Execution

The migration itself was comprised of several steps:

1. Deploying Migration Toolkit for Virtualization on the OpenShift cluster that we migrated to. Migration Toolkit for Virtualization can migrate VMs to the OpenShift cluster that it runs on or to remote OpenShift clusters that are added as destination providers. In our case, all the VMs were migrated to a single OpenShift cluster, so we installed the Migration Toolkit for Virtualization operator on that cluster.
2. In the OpenShift web console, we used an administrator user to configure the OpenShift cluster for the migration by:
   1. Creating a management project designated for the migration effort.
   2. Creating the Red Hat Virtualization source provider, the local OpenShift cluster as a target provider, and storage and network mappings in the management project.
3. Due to the large  number of migration plans that were required to migrate to the target projects, we automated their creation and execution using scripts we implemented that ran with administrator privileges. These scripts performed the following actions:
   1. Created a cold migration plan per target project. We chose cold migration because we were not concerned with the downtime of the VMs, and cold migrations are typically faster than warm migrations.
   2. Created a migration-by-migration plan to trigger the migration plans.

Most of the VM migration plans completed successfully, but we faced two interesting challenges during the VM migrations.

First, while we waited for the first VM migration to complete, we noticed that they took longer than expected. That was a consequence of executing a high number of migration plans simultaneously.

In Migration Toolkit for Virtualization, we can limit the number of VM migrations that are executed in parallel as part of a migration plan, but there is no way to limit the number of VM migrations that are executed in different migration plans. Thus, by running many migration plans simultaneously, we exhausted the network bandwidth, leading to slower VM migrations.

In our case, the time it took to migrate all the migration plans was more important than the downtime of each migrated VM, and the overall time of the migration was not affected by this choice. However, when the overall time of the migration is affected by exhausting the network bandwidth, which may happen when VM migrations time out or if the downtime of each VM needs to be minimized, you might want to either implement migration plans gradually or perform warm migrations.

Second, 3 out of 120 VM migrations failed. Our investigation of these failures found that they failed due to a bug in our codebase. We managed to fix the bug quickly and with a fixed version of Migration Toolkit for Virtualization, we successfully migrated all the VMs. The fix for this issue is included in Migration Toolkit for Virtualization 2.5.

With these adjustments, we successfully migrated all VMs to OpenShift.

Validating VMs on OpenShift

Once all the migrations completed successfully with Migration Toolkit for Virtualization, we performed sanity checks on  several randomly selected VMs on OpenShift Virtualization, with each selected VM passing successfully. Then we waited for more feedback from VM owners. Overall, we got positive feedback from users on their migrated VMs in OpenShift Virtualization, but a few issues were reported.

One issue involved some VMs that failed to boot, displaying a "no operating system found" message. We investigated and discovered that this happened only for VMs with multiple disks and only when an incorrect disk was chosen to boot from. We solved this problem by modifying the boot order of the VMs manually, making sure they would boot from their bootable disk. This issue is properly fixed in Migration Toolkit for Virtualization 2.5.

Also, a few service disruptions were reported. During the migration, VMs changed their VLAN due to infrastructure constraints, and some workloads and services within these VMs were inaccessible when they started. That was a consequence of having DNS records that were configured with different fully-qualified domain names (FQDNs) and IP addresses on the new VLAN. This problem was resolved by directing users to adapt their workloads and external clients to the new FQDN settings.

Lessons learned

Transitioning from traditional data centers to hybrid cloud solutions is challenging. There are various approaches available, supported by different tools, that users can choose from. In this article, we described a migration of VMs from Red Hat Virtualization to OpenShift that we did internally at Red Hat.

This migration required careful and comprehensive planning and clear communication with all stakeholders in the organization. While Migration Toolkit for Virtualization provides tooling for facilitating migrations to OpenShift Virtualization, it doesn’t always cover every technical step that is needed.

In this article, we shared how we used Migration Toolkit for Virtualization together with scripts we implemented for operations that are not handled by Migration Toolkit for Virtualization. These scripts are available at GitHub. With these tools, we managed to effectively migrate 120 VMs with 12 TB of data, which now run successfully on OpenShift.

As Red Hat Virtualization approaches its end-of-life, we expect many organizations to consider a similar migration. We hope that our experience and tools will help  you migrate VMs to OpenShift using  Migration Toolkit for Virtualization, which includes fixes for the issues mentioned in this post and brings many additional improvements.