Running Windows 11 and 2022 Server Virtual Machines in Red Hat OpenShift with persistent vTPM

The trusted platform module (TPM) is a self-contained hardware encryption technology present in recent computer systems. It provides, among other things, hardware random number generation and more secure storage for encryption keys. This enables administrators to encrypt operating system disks that will then only be decryptable on the same system. Version 2.0 of the TPM specification was published in 2015, and Microsoft’s Windows 11 requires a version 2.0 TPM to be present to install.

To support operating systems like Windows 11 that require a TPM, libvirt provides a virtual TPM (vTPM) that can be configured with a virtual machine (VM) to provide the appearance of a hardware TPM. Red Hat OpenShift Virtualization has supported vTPM as an option since Red Hat OpenShift 4.13, with the persistent storage capability added in OpenShift 4.14.

This means that a Windows 11 VM can employ BitLocker encryption to its system drives and OpenShift Virtualization will handle the encryption keys on its behalf through the vTPM interface. The virtualized TPM comes with an important caveat; it does not provide the same level of security as a physical chip. Any compliance controls that rely on that physical security should be re-evaluated when considering a move to virtual.

Preparing the cluster to support a persisted vTPM

To support the persistent storage of secrets in a VM’s vTPM, OpenShift Virtualization must create a Persistent Volume Claim (PVC) to back that secret storage. The storage class that provides vTPM state must be configured in the HyperConverged custom resource, typically “kubevirt-hyperconverged” in the “openshift-cnv” namespace. The storage class must be a “Filesystem” type (FS), and it should support the “ReadWriteMany” access mode (RWX) in order to allow VMs using a persistent vTPM to make use of live migration. An example of the parameter is below, taken from a cluster using Red Hat OpenShift Data Foundation:

spec:
   vmStateStorageClass: ocs-external-storagecluster-cephfs

Adding persistence to a VM

The change to the VM YAML to set the vTPM to persistent is likewise simple, if deeper in the hierarchy due to the spec.template.spec structure of a VM. The setting for TPM can be found under

spec.template.spec.domain.devices.tpm

To make the vTPM use persistent storage, add “persistent: true” under “tpm”:

oc patch vm win11 --type json -p '[{"op": "add", "path": "/spec/template/spec/domain/devices/tpm", "value": {"persistent": true}}]'

The change to the VM may be made while it is running, but it will not take effect until it is stopped and restarted by OpenShift Virtualization. For best results, include this parameter before running the VM for the first time.

Check that the TPM is now persisted by looking for its PVC. In this example, the VM “win11-broken-cow” has a persistent TPM:

$ oc get vm
NAME                AGE    STATUS    READY
win11-broken-cow    2d9h   Running   True
$ oc get pvc
NAME                                    STATUS     CAPACITY   ACCESS MODES
persistent-state-for-win11-broken-cow   Bound      12Mi       RWX
win11-broken-cow                        Bound      64Gi       RWX

(Some columns have been omitted from output for readability)

There are two important caveats around the vTPM feature. First, a non-persistent vTPM will store and return encryption keys during the lifetime of the virt-launcher Pod the VM runs on. This includes when the guest OS restarts. Therefore, in the case of BitLocker Drive Encryption for a Windows VM, it is not sufficient to select the option, “Run BitLocker system check” while going through the drive encryption wizard. The problem comes as soon as the virt-launcher Pod stops, which happens if you shut down or migrate the VM. Thus, it is possible to encrypt a VM’s disk then immediately lose access to that disk the next time it stops and starts. Windows does its best to force the user to save a copy of the recovery key; in this case, it would be the only way to recover the disk.

The second caveat is that the presence of a persistent vTPM disables the ability to snapshot the VM, as there is no mechanism for snapshotting the vTPM PVC as of the publication of this article. Because of this, online backups of VMs with persistent vTPMs will not be possible. The feature is being tracked in CNV-32718, currently slated for OpenShift 4.17.

Demonstrating persistence in a Windows 11 VM

Starting with a Windows 11 base image, create a new VM by cloning the Windows11 template. If there is not an existing base image, generate one by following this guide: Managing virtual machines with OpenShift Pipelines. Edit the YAML of the VM as shown above to include “persistent: true” under “tpm:”. Once the VM boots, check that a “persistent-state-for-” PVC exists for the chosen VM.

At this point, the C: drive may be encrypted with BitLocker Drive Encryption. Choose the option to print the recovery key to a PDF if using the wizard to start the encryption process as there is no currently no option with OpenShift Virtualization to attach a drive in a way Windows recognizes as removable, and the wizard will refuse to  continue if neither option is chosen. Another option for recovery is to join an Active Directory domain that has the BitLocker Key Recovery service set up, but this is beyond the scope of this article, and still insufficient to get the wizard to continue.

Once the encryption process completes, test the key persistence by shutting down the VM from OpenShift Virtualization. (Simply rebooting from within the OS will result in the OS being restarted in the same virt-launcher Pod it was previously running in, so is not a valid test of persistence.)

Conclusion

The addition of persistent vTPM to OpenShift Virtualization opens the door to compliant workloads that require encryption at rest. Work is planned to allow snapshotting of the persistent storage so encrypted VMs can be backed up, and the question of whether non-persisted vTPM should fail to save keys has been raised.