Back in December, we announced that Windows Container support for Red Hat OpenShift became generally available. This was made possible by the Windows MachineConfig Operator (WMCO), which acts as an entry point for OpenShift customers to run containerized Windows workloads in their OpenShift cluster. This Operator allows users to deploy a Windows worker node as a day 2 task. This gave users the ability to manage their Windows container workloads alongside Linux containers in AWS or Azure.

With OpenShift 4.7, we’ve made an update to the WMCO that now extends functionality and support to vSphere IPI clusters. Using the updated version WMCO Operator on OpenShift 4.7 or later, users can now run their Windows container workloads and Linux containers on one platform, vSphere.

In this blog, I will go over how to install and deploy a Windows worker node on OpenShift.

Prerequisites

A full list of prerequisites for installing on vSphere can be found in the official documentation. Please review these before attempting to install vSphere. Specifically, you’ll need the information about vCenter, a user with the right permissions, and two DNS entries. Again, consult the documentation if you have any questions.

Additionally, for Windows containers, you will need to install the OVNKubernetes network plug-in with Hybrid Networking Overlay customizations. It’s important to note that this can only be done at installation time*.

Once you’ve taken a look at the prerequisites, you can install OpenShift to VMware vSphere version 6.7U3 or 7.0 using the full-stack automation, also known as Installer Provisioned Infrastructure or IPI, experience.

* Since you cannot yet migrate between OpenShiftSDN and OVNKubernetes, only install with the OVN plus Hybrid networking overlay that has been upgraded to 4.7. and new 4.7 installs that are supported with Windows Containers.

You must also prepare a Windows Server 2019 “golden image” to use as a base of your Windows worker nodes. This includes installing all the latest updates, installing the docker runtime, installing and configuring OpenSSH, and installing the VMware tools.

Installing OpenShift With Hybrid Overlay Networking

You must install OpenShift using vSphere IPI with OVN and configure hybrid-overlay networking customizations. These steps are outlined in the official OpenShift documentation site. The Windows MachineConfig Operator will not work unless you install the cluster the way outlined in the documentation.

Once you have followed the documentation and the installer has finished the installation process, verify that hybrid overlay networking is enabled:

$ oc get network.operator cluster -o jsonpath='{.spec.defaultNetwork}' | jq -r
{
  "ovnKubernetesConfig": {
    "genevePort": 6081,
    "hybridOverlayConfig": {
      "hybridClusterNetwork": [
        {
          "cidr": "10.132.0.0/14",
          "hostPrefix": 23
        }
      ]
    },
    "mtu": 1400
  },
  "type": "OVNKubernetes"
}

Nnote that the OpenShift installer creates a directory inside of vSphere with the name of your cluster and a unique identifier:

This is important because we will be placing the Windows Server golden image in this folder.

Creating a Windows Server Golden Image

Before you can deploy a Windows worker, you first need to prepare a VM template to be used by the OpenShift Machine API to build a Windows worker. Moreover, the WMCO uses SSH in order to install and configure the Windows machine as a worker. Therefore, we’ll need to first create an SSH key for the WMCO to use:.

$ ssh-keygen -t rsa -b 4096 -N '' -f ${HOME}/.ssh/windows_node

Once you’ve created the SSH key, you can proceed to prepare your Windows node. To prepare the Windows, you’ll need the following:

• Windows Server 2019 LTSC version 10.0.17763.1457 or older.
• The following patch must be installed - KB4565351.
• Open TCP port 10250 on the Windows firewall for container logs.
• SSH installed with key-based authentication as the Administrator user with the keypair you created. This is usually done by adding the public key to the authorized_keys file into the .ssh directory in the Administrator’s directory. Currently, using the local Administrator account is required and the host must not be joined to a domain.
• Docker runtime installed.
• Any base images pre-pulled into the server. See this matrix for compatibility.
• VMware Tools installed. Note that the C:\ProgramData\VMware\VMware Tools\tools.conf file must contain exclude-nics= to make sure the vNIC generated on the Windows VM by the hybrid overlay won't be ignored.
• The server sysprep-ed to where SSH key based authentication is preserved.

For consistency, name the Windows VM <clustername>-<id>-windows. This is consistent with how the Red Hat Enterprise Linux CoreOS (RHCOS) VM template is named. Save this VM in the same folder as the one your cluster is running in. In the end, you’ll see something like this, below. (I’ve outlined the Windows VM in a red box.)

After you’ve run the sysprep command and shutdown the VM, you can leave it as a VM (which is what the RHCOS VM is) or convert it into a template. The WMCO will work with both.

Installing the Windows MachineConfig Operator

The WMCO can be installed via OperatorHub on your cluster. This can easily be done via the OpenShift UI:.

Please consult the official documentation for specifics on how to install the Windows MachineConfig Operator. The high-level steps are outlined below:

• From the administrator’s perspective, use the left side navigation to go to Operators → OperatorHub. 
• In the Filter by keyword type in Windows Machine Config Operator to filter the catalog, then select the Windows Machine Config Operator tile.
• After reviewing the information, go ahead and click Install.
• On the Install Operator page: Select 4.7 as the update channel, leave the Installation Mode and Installed Namespace to their respective defaults. Select either Automatic or Manual for the Approval Strategy.
• Click Install.

After a while, you should see a Pod in the openshift-windows-machine-config-operator namespace:

$ oc get pods -n openshift-windows-machine-config-operator
NAME                                               READY   STATUS    RESTARTS   AGE
windows-machine-config-operator-7747fcd446-x2z27   1/1     Running   2          58s

The WMCO uses SSH to interact with the Windows machine. You will need to load the SSH private key into a secret. This is the same key that you created when you prepared the Windows Server golden image.

Load the private key as a secret into the openshift-windows-machine-config-operator namespace:

$ oc create secret generic cloud-private-key \
--from-file=private-key.pem=${HOME}/.ssh/windows_node \ 
-n openshift-windows-machine-config-operator

The key should now be loaded into OpenShift:

$ oc get secret cloud-private-key -n openshift-windows-machine-config-operator
NAME                TYPE     DATA   AGE
cloud-private-key   Opaque   1      2d2h

Deploying a Windows Machine

Creating a Windows Machine uses the same process as creating a RHCOS Machine using a MachineSet object. To build a MachineSet YAML file, you’ll need the clusterID. You can get this with the following command:

$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster
openshift4-9r24l

Using this, and information about my vSphere cluster and folder/VM names, I can build my MachineSet:

apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata:
  labels:
    machine.openshift.io/cluster-api-cluster: openshift4-9r24l
  name: winmach
  namespace: openshift-machine-api
spec:
  replicas: 1
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-cluster: openshift4-9r24l
      machine.openshift.io/cluster-api-machineset: winmach
  template:
    metadata:
      labels:
        machine.openshift.io/cluster-api-cluster: openshift4-9r24l
        machine.openshift.io/cluster-api-machine-role: worker
        machine.openshift.io/cluster-api-machine-type: worker
        machine.openshift.io/cluster-api-machineset: winmach
        machine.openshift.io/os-id: Windows
    spec:
      metadata:
        labels:
          node-role.kubernetes.io/worker: ""
      providerSpec:
        value:
          apiVersion: vsphereprovider.openshift.io/v1beta1
          credentialsSecret:
            name: vsphere-cloud-credentials
          diskGiB: 128
          kind: VSphereMachineProviderSpec
          memoryMiB: 16384
          network:
            devices:
            - networkName: VM Network
          numCPUs: 4
          numCoresPerSocket: 1
          snapshot: ""
          template: /Datacenter/vm/openshift4-9r24l/openshift4-9r24l-windows
          userDataSecret:
            name: windows-user-data
          workspace:
             datacenter: Datacenter
             datastore: datastore1
             folder: /Datacenter/vm/openshift4-9r24l
             resourcePool: /Datacenter/host/homecluster/Resources
             server: vsphere.example.com

The full list of options and values available for the MachineSet YAML can be seen by visiting the official documentation page on how to create it. I will go over the import sections.

• Every label key of machine.openshift.io/cluster-api-cluster must have the value of the clusterID, which in this case is openshift4-9r24l.
• The .metadata.name is the name you want your MachineSet to have. Please note that the MachineSet name cannot be more than nine characters long, due to the way machine names are generated in vSphere.
• Every label key of machine.openshift.io/cluster-api-machineset must have the value of the MachineSet name you chose.
• Note that the userDataSecret is set to windows-user-data. This secret is generated by the WMCO Operator for you.
• Note that the template value is set as a full path to the VM you ran sysprep on.
• The workspace section is information about your vSphere cluster OpenShift installation. This can be found in your current MachineSets by running the following: oc get machinesets -n openshift-machine-api -o yaml

Once you’ve set up your MachineConfig YAML file (I’ve named mine windows-ms.yaml) apply it to your cluster:

$ oc apply -f windows-ms.yaml

Once this YAML is applied, you can inspect the operator logs to see the WMCO create, bootstrap, and install the Windows Node:.

$ oc logs -f -l name=windows-machine-config-operator -n openshift-windows-machine-config-operator

You can see that this creates a MachineSet for you:

$ oc get machinesets -n openshift-machine-api
NAME                          DESIRED   CURRENT   READY   AVAILABLE   AGE
openshift4-9r24l-worker   3             3             3           3               3d4h
winmach                       1             1                                 13s

$ oc get machines -n openshift-machine-api
NAME                                PHASE              TYPE   REGION   ZONE   AGE
openshift4-9r24l-master-0           Running                                   3d4h
openshift4-9r24l-master-1           Running                                   3d4h
openshift4-9r24l-master-2           Running                                   3d4h
openshift4-9r24l-worker-b28k9   Running                                   3d4h
openshift4-9r24l-worker-csbfw   Running                                   3d4h
openshift4-9r24l-worker-gqkbt   Running                                   3d4h
winmach-8t6dd                       Provisioning                              92s

After a while, this MachineSet, winmach-8t6dd, wil go from “Provisioning” to “Provisioned”:

$ oc get machine winmach-8t6dd -n openshift-machine-api
NAME                PHASE             TYPE   REGION   ZONE   AGE
winmach-8t6dd   Provisioned                              3m15s

• Transfers required binaries to set up the Windows worker.
• Remotely configures the kubelet.
• Installs and configures the hybrid overlay networking.
• Configures the CNI.
• Sets up and manages the kube-proxy process.
• Approves the CSRs so the node can join the cluster.

Soon  the Windows Machine should be configured as an OpenShift worker:

$ oc get nodes -l kubernetes.io/os=windows

NAME                STATUS   ROLES        AGE   VERSION

winmach-8t6dd   Ready        worker   25m   v1.20.0-1030+cac2421340a449

If you run an oc describe node winmach-8t6dd you can see more information about the Windows node. Here is a snippet that shows the System Info:

System Info:
  Machine ID:                     winmach-8t6dd
  System UUID:                    BD383E42-86C4-D91B-C667-281EBCD42025
  Boot ID:                        
  Kernel Version:                 10.0.18363.778
  OS Image:                       Windows Server Datacenter
  Operating System:               windows
  Architecture:                   amd64
  Container Runtime Version:  docker://19.3.14
  Kubelet Version:                v1.20.0-1030+cac2421340a449
  Kube-Proxy Version:             v1.20.0-1030+cac2421340a449

Please note that this host has a Taint. This is to isolate Windows workloads and not let the scheduler try and schedule Linux containers (which would fail anyway) on this host:

$ oc describe node winmach-8t6dd | grep Taint
Taints:                 os=Windows:NoSchedule

To login to this node, I’ll first need to know it’s IP address. I can obtain the IP address of my Windows node with the following command:

$ oc get  node winmach-8t6dd -o jsonpath='{.status.addresses}' | jq -r
[
  {
        "address": "192.168.1.16",
        "type": "ExternalIP"
  },
  {
        "address": "192.168.1.16",
        "type": "InternalIP"
  },
  {
        "address": "winmach-8t6dd",
        "type": "Hostname"
  }
]

In this output, we want the ExternalIP. You can SSH into this Windows worker as the user Administrator with the SSH key you created earlier:

$ ssh  -i ${HOME}/.ssh/windows_node Administrator@192.168.1.16 powershell

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS C:\Users\Administrator> 

Verify that docker is running:

PS C:\Users\Administrator> docker ps

CONTAINER ID    IMAGE    COMMAND    CREATED   STATUS     PORTS   NAMES 

PS C:\Users\Administrator> Get-Process | ?{ $_.ProcessName -match "kube|overlay|docker" }
Handles  NPM(K)        PM(K)          WS(K)      CPU(s)         Id  SI ProcessName
-------  ------        -----          -----      ------         --  -- -----------
273          18        66068          42812        9.92   1756   0 dockerd
201          17        30296          37092        6.28   1920   0 hybrid-overlay-node
331          25        52656          76492       97.41   4580   0 kubelet
242          19        33076          43468       29.81   4332   0 kube-proxy

Go ahead and exit this session by typing exit in the PowerShell session.

Scaling a Windows MachineSet

The paradigm used to interact with the Windows MachineSet is the same as the RHCOS MachineSets. This includes creating, managing, and scaling. To scale the Windows MachineSet, you’d run the following command:

$ oc scale machineset winworker --replicas=2 -n openshift-machine-api

After some time, you should see two Windows nodes in your cluster:

$ oc get nodes -l kubernetes.io/os=windows
NAME              STATUS   ROLES    AGE     VERSION
winmach-8t6dd   Ready    worker   2m39s   v1.20.0-1030+cac2421340a449
winmach-s5kmd   Ready    worker   12m     v1.20.0-1030+cac2421340a449

You should see two new “winmach” machines as well:

$ oc get machines -n openshift-machine-api \
-l machine.openshift.io/cluster-api-machineset=winmach
NAME              PHASE     TYPE   REGION   ZONE   AGE
winmach-8t6dd   Running                          5m49s
winmach-s5kmd   Running                          14m

Deploying Sample Workload

With the Windows MachineConfig Operator and Windows worker in place, you can now deploy an application. Here is an example of the application I’m going to deploy:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: win-webserver
  name: win-webserver
  namespace: windows-workloads
spec:
  selector:
    matchLabels:
      app: win-webserver
  replicas: 1
  template:
    metadata:
      labels:
        app: win-webserver
      name: win-webserver
    spec:
      tolerations:
      - key: "os"
        value: "Windows"
        Effect: "NoSchedule"
      containers:
      - name: windowswebserver
        image: mcr.microsoft.com/powershell:lts-nanoserver-1909
        imagePullPolicy: IfNotPresent
        command:
        - pwsh.exe
        - -command
        - $listener = New-Object System.Net.HttpListener; $listener.Prefixes.Add('http://*:80/'); $listener.Start();Write-Host('Listening at http://*:80/'); while ($listener.IsListening) { $context = $listener.GetContext(); $response = $context.Response; $content='<html><body><H1>OpenShift for Windows Containers</H1></body></html>'; $buffer = [System.Text.Encoding]::UTF8.GetBytes($content); $response.ContentLength64 = $buffer.Length; $response.OutputStream.Write($buffer, 0, $buffer.Length); $response.Close(); };
        securityContext:
          windowsOptions:
            runAsUserName: "ContainerAdministrator"
      nodeSelector:
        kubernetes.io/os: windows

A few things to note here:

• I have Tolerations in place so that this Deployment will be able to be scheduled on the Windows worker node.
• I am using the mcr.microsoft.com/powershell:lts-nanoserver-1909 image. The image is specific to the version of Windows you run. Please consult the compatibility matrix to determine which image is best for your installation.
• The “ContainerAdminsitrator” under the securityContext section indicates which user inside the container will run the process. For more information, please see this doc.
• Lastly, note that I am using a nodeSelector of kubernetes.io/os: windows. This will place this Deployment on the Windows node since the node is labeled as such.

To apply this Deployment, first you need to create a namespace. This sample application expects a namespace of windows-workloads:

$ oc new-project   windows-workloads
Now using project "windows-workloads" on server "https://api.openshift4.example.com:6443".

$ oc apply -f sample-winc-app.yaml
deployment.apps/win-webserver created

Then you should see the Pod running on your Windows node:

$ oc get pods -o wide
NAME                               READY   STATUS        RESTARTS   AGE         IP               NODE                NOMINATED NODE   READINESS GATES
win-webserver-95584c8c-7dssq   1/1         Running   0              3m31s   10.132.0.2   winmach-8t6dd   <none>               <none>

$ oc expose deployment win-webserver --target-port=80 --port=8080
service/win-webserver exposed

This should have created a service of type ClusterIP:

$ oc get svc
NAME                TYPE            CLUSTER-IP          EXTERNAL-IP   PORT(S)        AGE
win-webserver   ClusterIP   172.30.147.30   <none>            8080/TCP   69s

Let’s now create a route by exposing this service:

$ oc expose svc win-webserver
route.route.openshift.io/win-webserver exposed

You can now visit this application by viewing the route on your web browser. To get the route, run the following command:

$ oc get route win-webserver -o jsonpath='{.spec.host}{"\n"}'
win-webserver-windows-workloads.apps.openshift4.example.com

The page should display the following:

Application Logging

Getting the logging output from a Windows container is the same as a Linux container. You can use the same command. For example:

$ oc logs -f win-webserver-95584c8c-7dssq  -n windows-workloads
Listening at http://*:80/

If you need to debug further, you can even run an oc rsh into the container to start an interactive PowerShell session:

$ oc -n windows-workloads rsh win-webserver-95584c8c-7dssq pwsh
PowerShell 7.0.5
Copyright (c) Microsoft Corporation. All rights reserved.
https://aka.ms/powershell
Type 'help' to get help.
PS C:\>

Notice that I passed pwsh since the default shell for oc rsh is bash. This way you can get an interactive PowerShell session inside the container.

Application Storage

You can use the in-tree vSphere volume plug-in for storage. You must use storage outside the PV/PVC paradigm as the CSI vSphere plugin is not currently supported*. 

* Note: CSI-proxy is still in alpha status. More info here.

To use the in-tree volume plug-in, first login to an ESXi host via SSH and create the storage:

[root@esxi:~] vmkfstools -c 2G /vmfs/volumes/datastore1/myDisk.vmdk

Note that here, you need to specify the path to your datastore. In my case, my datastore is named datastore1. For more information, please see the upstream Kuberentes documentation.

You can specify the volume in your container definition in your deployment or pod. Here is an example of using this in a Pod:

apiVersion: v1
kind: Pod
metadata:
  name: win-storage
spec:
  securityContext:
    windowsOptions:
      runAsUserName: "ContainerAdministrator"
  nodeSelector:
    kubernetes.io/os: windows
  tolerations:
  - key: "os"
    value: "Windows"
    Effect: "NoSchedule"
  containers:
  - name: windowswebserver
    image: mcr.microsoft.com/powershell:lts-nanoserver-1909
    imagePullPolicy: IfNotPresent
    command:
    - pwsh.exe
    - -command
    - $listener = New-Object System.Net.HttpListener; $listener.Prefixes.Add('http://*:80/'); $listener.Start();Write-Host('Listening at http://*:80/'); while ($listener.IsListening) { $context = $listener.GetContext(); $response = $context.Response; $content='<html><body><H1>OpenShift for Windows Containers</H1></body></html>'; $buffer = [System.Text.Encoding]::UTF8.GetBytes($content); $response.ContentLength64 = $buffer.Length; $response.OutputStream.Write($buffer, 0, $buffer.Length); $response.Close(); };
    volumeMounts:
    - mountPath: c:\Data
      name: test-volume
  volumes:
  - name: test-volume
    vsphereVolume:
      volumePath: "[datastore1] myDisk"
      fsType: ntfs

This is similar to the container definition inside the deployment of the application we first deployed, but I do want to call out a few things.Here in the .spec.volumes I am specifying the datastore and the name of the disk. I’m also specifying the fsType to be ntfs. Additionally, in the .spec.containers.volumeMounts I am setting the mountPath to c:\Data.

Let’s deploy this sample pod:

$ oc apply  -f sample-pod.yaml

Once the pod is up and running, you can rsh into the pod to see the storage:

$ oc rsh win-storage pwsh
PowerShell 7.0.5
Copyright (c) Microsoft Corporation. All rights reserved.
https://aka.ms/powershell
Type 'help' to get help.
PS C:\>

Here you can now use the storage inside your pod:

PS C:\> cd c:\data
PS C:\data> echo "test" > testfile.txt
PS C:\data> dir
    Directory: C:\data
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---           2/13/2021  1:39 PM              6 testfile.txt

If you SSH into the Windows VM, you can see the disk by running Get-Disk in a PowerShell session. You will see that the 2G volume is mounted on the Windows VM host:

PS C:\Users\Administrator> Get-Disk
Number Friendly Name           Serial Number                    HealthStatus         OperationalStatus      Total Size Partition
                                                                                                                       Style
------ -------------           -------------                    ------------         -----------------      ---------- ----------
1      VMware Virtual disk     6000c291798c8f60847975e552661843 Healthy              Online                       2 GB MBR
0      VMware Virtual disk     6000c29804a9a62ad295b0f130e0e7dc Healthy              Online                     128 GB GPT

Conclusion

This post has covered getting started with deploying Windows nodes to your OpenShift cluster and deploying a sample application to verify functionality. Containers, Kubernetes, and OpenShift have almost exclusively been the domain of Linux since their inception, but many organizations have Windows-only applications which can benefit from containers, too.  OpenShift 4.7 with the Windows MachineConfig Operator enables our customers to deploy Windows and Linux containers to the same cluster, simplifying development and deployment processes and consolidating administration to one platform, OpenShift.

I have just scratched the surface of what's possible here.There is a whole world of possibilities opened up with this new feature, including integrating Windows containers into your OpenShift- based CI/CD processes, autoscaling .NET applications based on CPU, memory, and custom metrics, or just containerizing that one, last, stubborn microservice.

For more information, see the official docs, OpenShift TV, and our other blog posts. I did a stream about this on the OpenShift Ask The Admin show; which you can watch the recording of here.