User defined networks in Red Hat OpenShift Virtualization

One of the most anticipated features in the Red Hat OpenShift 4.18 release is the ability to use discrete user defined networks in OpenShift Virtualization. Prior to the introduction of this feature, the only options for networking were the shared OpenShift pod network, layer-2 unroutable network attachment definitions within a namespace, or configuring virtual machines to use external networks. 

With user defined networks, you now have the ability to create private networks within your own namespace or even across multiple namespaces that are segregated from other networks. This feature affords you the flexibility to connect your virtual machines (VM) together in the manner that best suits your organization’s needs.

Details for creating and managing user defined networks for OpenShift Virtualization can be found in our official documentation. In this article, I provide specific working examples you can use in your environment.

Types of user defined networks

There are two types of user defined network custom resources (CR), discussed below:

• UserDefinedNetwork
• ClusterUserDefinedNetwork

UserDefinedNetwork

A UserDefinedNetwork (UDN) is a CR that provides network isolation between tenants within the same namespace. In other words, when you create a UserDefinedNetwork, you are creating a network with an arbitrary subnet that your VMs use to communicate with each other. This network namespace is segregated from other networks and may only be used by the VMs you provide in that namespace.

As shown in the diagram above, you may have one UserDefinedNetwork per namespace and only VMs in that namespace may communicate on that UDN. VMs on the udn-1 UDN are not able to communicate with VMs on the other UDNs. This provides total isolation between network tenants and namespaces.

The steps to create a UserDefinedNetwork are:

• Create a namespace with the appropriate label
• Create the UserDefinedNetwork
• Create the Virtual Machine

Creating a Namespace for the UserDefinedNetwork

Creating a UserDefinedNetwork in a namespace effectively replaces the pod network with the UDN that you create. As a result, once the UDN is created, your VMs cannot access the pod network.

To start, the namespace in which you intend to create the UDN must have the following label applied:

k8s.ovn.org/primary-user-defined-network=""

Without this label, the UDN fails to create. Furthermore, the label must be applied when the namespace is created. It is not possible to create the namespace first, and then apply the label afterward. If you attempt to apply the label after the namespace has been created, you will receive an error message and the command fails:

$ oc label namespace/vmtest k8s.ovn.org/primary-user-defined-network=""
The namespaces "vmtest" is invalid: : ValidatingAdmissionPolicy 'user-defined-networks-namespace-label' with binding 'user-defined-networks-namespace-label-binding' denied request:
The 'k8s.ovn.org/primary-user-defined-network' label cannot be added/removed after the namespace was created

To create the namespace with the label, use the OpenShift command line with a YAML file. Our namespace in this example is udn-dev:

$ cat udn-dev-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: udn-dev
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
$ oc create -f udn-dev-ns.yaml
namespace/udn-dev created

Create the UserDefinedNetwork

Now that the namespace has been created properly, you can create the UserDefinedNetwork using a YAML file:

$ cat udn-dev.yaml
apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: udn-dev
  namespace: udn-dev
spec:
  layer2:
    ipam:
      lifecycle: Persistent
    role: Primary
    subnets:
    - 10.200.0.0/16
  topology: Layer2

In the example file above, we have defined a layer-2 UDN with the name udn-dev and a subnet of 10.200.0.0/16. You may use whatever unique name you like for your UDN and you may also use any subnet configuration. Since UDNs are segregated from each other, there is no concern about CIDR overlap between namespaces. In other words, having a UDN with a CIDR of 10.200.0.0/16 in one namespace and 10.200.10.0/24 in another namespace is a valid configuration because the two namespaces will never communicate with each other.

In the example above, the IPAM (IP Address Management) is set to Persistent. This ensures that the VM  retains the same IP address after a reboot.

Once you have named your UDN and set the subnet CIDR per your preferences, you may create the UDN from a YAML file:

$ oc create -f udn-dev.yaml
userdefinednetwork.k8s.ovn.org/udn-dev created
The UDN creation completes very quickly and you are able to examine the resulting CR:
$ oc get userdefinednetwork -n udn-dev
NAME     AGE
udn-dev  30s

After the UDN is created, view the details to confirm that there were no errors during its creation and to validate the status:

$ oc get userdefinednetwork/udn-dev -o yaml -n udn-dev
apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  creationTimestamp: "2025-02-26T21:55:54Z"
  finalizers:
  - k8s.ovn.org/user-defined-network-protection
  generation: 1
  name: udn-dev
  namespace: udn-dev
  resourceVersion: "1162536"
  uid: 035f9159-d02e-44ed-b61f-14d73d2558f7
spec:
  layer2:
    ipam:
      lifecycle: Persistent
    role: Primary
    subnets:
    - 10.200.0.0/16
  topology: Layer2
status:
  conditions:
  - lastTransitionTime: "2025-02-26T21:55:55Z"
    message: NetworkAttachmentDefinition has been created
    reason: NetworkAttachmentDefinitionCreated
    status: "True"
    type: NetworkCreated
  - lastTransitionTime: "2025-02-26T21:55:55Z"
    message: Network allocation succeeded for all synced nodes.
    reason: NetworkAllocationSucceeded
    status: "True"
    type: NetworkAllocationSucceeded

Pay special attention to the status: portion of the output above as it contains critical information on the status of the UDN. The first message we want to see is the last one that indicates:

message: Network allocation succeeded for all synced nodes.

This tells you that the UDN was created successfully. The other message tells you that the NetworkAttachmentDefinition for this UDN was created successfully as well:

message: NetworkAttachmentDefinition has been created

The NetworkAttachmentDefinition is automatically used by any VMs created in this namespace. Although you don't need to interact with the NetworkAttachmentDefinition directly, it's good to validate its presence and name:

$ oc get network-attachment-definitions -n udn-dev
NAME       AGE
udn-dev    107s

Create a virtual machine on the UserDefinedNetwork

Now that the Namespace and UserDefinedNetwork are created, we are ready to create a virtual machine. The detailed steps to create a virtual machine are out of scope of this article, but you can find instructions for creating a virtual machine in another one of my blog posts. For this article, I demonstrate how to create a simple virtual machine with no customizations.

Because the namespace in which this virtual machine is created has the  k8s.ovn.org/primary-user-defined-network: ""  label, the VM is automatically configured to  use the UDN we created earlier.

In your environment, feel free to customize your virtual machine, but importantly, do not modify the network configuration. Because the namespace is already configured to use user defined networks, the virtual machine network is already configured appropriately, and any modifications may cause the virtual machine to not connect to the UDN.

Once the virtual machine boots, you can login and see that the network interface has received an IP address from DHCP in the CIDR specified in the UserDefinedNetwork configuration, as well as a router and access to external networks. When using user defined networks, it is important to enable a DHCP client on the virtual machine’s NIC. Attempting to set a different IP address manually, even if it is on the proper network, doesn't permit network traffic. Although you may hard-code the IP address that is presented via DHCP, this is not recommended as the DHCP lease expiring and being renewed with a different IP could cause an interruption in network connectivity.

The network interface has received an IP address of 10.200.0.3 from the DHCP server on the UDN:

[rhel@rhel9-udndev-01 ~]$ ip address show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1342 qdisc fq_codel state UP group default qlen 1000
    link/ether 0a:58:0a:c8:00:03 brd ff:ff:ff:ff:ff:ff
    inet 10.200.0.3/16 brd 10.200.255.255 scope global dynamic noprefixroute eth0
      valid_lft 3039sec preferred_lft 3039sec

Despite the UDN being a Layer-2 network, a software-defined router is available on the network:

[rhel@rhel9-udndev-01 ~]$ ip route
default via 10.200.0.1 dev eth0 proto dhcp src 10.200.0.3 metric 100
10.200.0.0/16 dev eth0 proto kernel scope link src 10.200.0.3 metric 100

As a result, the VM is able to communicate with external resources:

[rhel@rhel9-udndev-01 ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=113 time=7.06 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=113 time=6.02 ms

Any virtual machine created in this namespace will have a NIC on the UDN and will be able to communicate with other VMs in the namespace.

ClusterUserDefinedNetwork

A ClusterUserDefinedNetwork is similar to a UserDefinedNetwork except that it can be used across namespaces. In other words, with a ClusterUserDefined Network (cluster UDN), VMs in different namespaces can communicate with each other via the cluster UDN. Any VMs using the ClusterUserDefinedNetwork are segregated from other VMs so long as they are not using the same cluster UDN, regardless of their namespace. ClusterUserDefinedNetworks may only be created by a cluster administrator.

As shown in the diagram above, a cluster UDN may span more than one namespace, allowing VMs in different namespaces the ability to communicate with one another while maintaining network isolation from other VMs that are not connected to the same cluster UDN.

Create a namespace for the ClusterUserDefinedNetwork

As with UserDefinedNetworks, a namespace must be created with the appropriate label and that label may not be applied to an existing namespace. Therefore, is it easiest to create the namespace with the correct label using the OpenShift command line interface and a YAML file. In our example, we will create two namespaces and then create a cluster UDN that allows VMs to communicate between them.

First create the udn-prod namespace:

$ cat udn-prod.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: udn-prod
  labels:
    k8s.ovn.org/primary-user-defined-network: “”
$ oc create -f udn-prod.yaml
namespace/udn-prod created

Then create the udn-pre-prod namespace, which is identical except for the name:

$ cat udn-pre-prod.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: udn-pre-prod
  labels:
    k8s.ovn.org/primary-user-defined-network: “”
$ oc create -f udn-pre-prod.yaml
namespace/udn-pre-prod created

Before we create the ClusterUserDefinedNetwork, apply some additional labels to the new namespaces. Although this is not strictly required, it assists us later in selecting the namespaces when we define the cluster UDN. In this example, the cluster UDN is for our production servers, so we label the namespaces with an appropriately descriptive label. The label is arbitrary and can be whatever you want so long as it matches the NamespaceSelector in the cluster UDN definition later on. We use the label cluster-udn=prod to identify our production namespaces.

$ oc label namespace/udn-prod cluster-udn=prod
namespace/udn-prod labeled
$ oc label namespace/udn-pre-prod cluster-udn=prod
namespace/udn-pre-prod labeled

Create the UserDefinedNetwork

Now that the namespaces have been created and labeled properly, we can create the ClusterUserDefinedNetwork using a YAML file:

$ cat cluster-udn-prod.yaml
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  name: cluster-udn-prod
spec:
  namespaceSelector:
    matchLabels:
      cluster-udn: prod
  network:
    layer2:
      ipam:
        lifecycle: Persistent
      role: Primary
      subnets:
      - 10.100.0.0/16
    topology: Layer2

In the example file above, we have defined a layer-2 cluster UDN with the name cluster-udn-prod and a subnet of 10.100.0.0/16. You may use whatever name you like for your cluster UDN and you may also use any subnet configuration.  Importantly, we have also included a namespaceSelector that includes the label we gave to the namespaces earlier. This ensures that the cluster UDN is available to the appropriate namespaces once it's created.

In the example above, the IPAM (IP Address Management) is set to Persistent. This ensures that the VM retains the same IP address after a reboot.

Once you've named your cluster UDN and set the subnet CIDR, you can create the cluster UDN from a YAML file:

$ oc create -f cluster-udn-prod.yaml
clusteruserdefinednetwork.k8s.ovn.org/cluster-udn-prod created

After the cluster UDN is created, you can view the details to confirm that there were no errors during its creation and to validate the status:

$ oc get clusteruserdefinednetwork/cluster-udn-prod -o yaml
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  creationTimestamp: "2025-02-26T21:36:35Z"
  finalizers:
  - k8s.ovn.org/user-defined-network-protection
  generation: 1
  name: cluster-udn-prod
  resourceVersion: "1149053"
  uid: 3dfdb159-889d-4ac4-ae0b-44b65fe881f4
spec:
  namespaceSelector:
    matchLabels:
      cluster-udn: prod
  network:
    layer2:
      ipam:
        lifecycle: Persistent
      role: Primary
      subnets:
      - 10.100.0.0/16
    topology: Layer2
status:
  conditions:
  - lastTransitionTime: "2025-02-26T21:36:35Z"
    message: 'NetworkAttachmentDefinition has been created in following namespaces:
   [udn-pre-prod, udn-prod]'
    reason: NetworkAttachmentDefinitionCreated
    status: "True"
    type: NetworkCreated

The most important part of the output above is the text in the status: section. The type: field indicates NetworkCreated and the message: field indicates that a NetworkAttachmentDefinition has been created in each of the two namespaces that we had labeled: udn-pre-prod and udn-prod.

The NetworkAttachmentDefinitions are automatically used by any VMs created in the selected namespaces. Although we will not need to interact with the NetworkAttachmentDefinitions directly, it's good to validate their presence and names:

$ oc get network-attachment-definitions -A
NAMESPACE     NAME              AGE
udn-pre-prod cluster-udn-prod    22h
udn-prod     cluster-udn-prod    22h

One NetworkAttachmentDefinition was created for each namespace in which the cluster UDN was created. The new VMs in each namespace will use these NetworkAttachmentDefinitions for their primary network interfaces.

After the cluster UDN is created, you can add a new namespace to it by following the same process above to create the namespace with the appropriate label for enabling user defined networks, and then adding the cluster-udn: prod label to it. The new namespace will automatically have the cluster UDN extended to it once the appropriate labels are present.

Create virtual machines on the ClusterUserDefinedNetwork

Now that the namespaces and ClusterUserDefinedNetwork are created, we are ready to create virtual machines. The detailed steps to create a virtual machine are out of scope of this article, but you can find instructions for creating a virtual machine in another one of my blog posts. In this article, I demonstrate creating a simple virtual machine in the udn-prod namespace  with no customizations.

Because the namespace in which this virtual machine is created has the  k8s.ovn.org/primary-user-defined-network: ""  label, the VM is automatically configured to  use the Cluster UDN we created earlier.

In your environment, feel free to customize your virtual machines, but importantly, do not modify the Network configuration. Because the namespace is already configured to use user defined networks, the virtual machine network is already configured appropriately and any modifications may cause the virtual machines to not connect to the UDN.

After the virtual machine boots, you can log in and see that the network interface has an IP address in the CIDR specified in the UserDefinedNetwork configuration, as well as a router and access to external networks. When using user defined networks, it is important to enable a DHCP client on the virtual machine’s NIC. Attempting to set a different IP address manually, even if it is on the proper network, will not permit network traffic. Although you may hard-code the IP address that is presented via DHCP, this is not recommended as the DHCP lease expiring and being renewed with a different IP could cause an interruption in network connectivity.

The network interface has received an IP address of 10.100.0.3 from the DHCP server on the cluster UDN:

[rhel@rhel9-udnprod-01 ~]$ ip address show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1342 qdisc fq_codel state UP group default qlen 1000
    link/ether 0a:58:0a:64:00:03 brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.3/16 brd 10.100.255.255 scope global dynamic noprefixroute eth0
    valid_lft 1843sec preferred_lft 1843sec

Despite the UDN being a Layer-2 network, a software-defined router is available on the network:       

[rhel@rhel9-udnprod-01 ~]$ ip route
default via 10.100.0.1 dev eth0 proto dhcp src 10.100.0.3 metric 100
10.100.0.0/16 dev eth0 proto kernel scope link src 10.100.0.3 metric 100

As a result, the VM is able to communicate with external resources:

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=113 time=12.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=113 time=7.39 ms

Although the cluster UDN was defined as a Layer2 network, there is a router automatically created so that the VMs can access the external network.

The virtual machine above was created in the udn-prod namespace. After we create an identical VM in the udn-pre-prod namespace which shares the same cluster UDN as udn-prod, we will see that it is similarly configured and can communicate with the VM we created above through the cluster UDN.

The network interface on the new VM has received an IP address of 10.100.0.4 from the DHCP server on the cluster UDN:

[rhel@rhel9-udnpreprod-01 ~]$ ip address show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1342 qdisc fq_codel state UP group default qlen 1000
    link/ether 0a:58:0a:64:00:04 brd ff:ff:ff:ff:ff:ff
    inet 10.100.0.4/16 brd 10.100.255.255 scope global dynamic noprefixroute eth0
    valid_lft 2724sec preferred_lft 2724sec

The new VM in the udn-pre-prod namespace is able to communicate with the first VM in the udn-prod namespace because they share the same cluster UDN. As a reminder, the IP address of this VM in the udn-pre-prod namespace is 10.100.0.4 and the other VM on the same cluster UDN but in the udn-prod namespace is 10.100.0.3. You can see that communication across the namespaces is successful.

[rhel@rhel9-udnpreprod-01 ~]$ ping 10.100.0.3
PING 10.100.0.3 (10.100.0.3) 56(84) bytes of data.
64 bytes from 10.100.0.3: icmp_seq=1 ttl=64 time=4.97 ms
64 bytes from 10.100.0.3: icmp_seq=2 ttl=64 time=2.90 ms

Examine the virtual machine network definition

The steps above have shown you how to deploy and use user defined networks in OpenShift Virtualization. In the section below, we will examine the network configuration in the VMs we created above to demonstrate how the VM configuration differs when using UDN. The network definition for the VM is the same whether you are using UserDefinedNetworks or ClusterUserDefinedNetworks.

In the .spec.template.spec.domain.devices for the VM, there is a definition for our cluster UDN. Note that the value l2bridge is the only valid option for virtual machines at this time.

spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
            - binding:
                name: l2bridge
              name: default

Furthermore, the section .spec.template.spec.networks in the VM definition has a reference to the same interface, which in this case is named default:

spec:
  template:
    spec:
      networks:
        - name: default
          pod: {}

The name of the network and the name of the interface must match.

Conclusion

User defined networking for virtual machines is a highly anticipated new feature in OpenShift 4.18 that brings more flexibility to many OpenShift Virtualization users. As exciting as it is, Red Hat is planning even more flexibility and use cases for user defined networks in OpenShift 4.19 and 4.20.

User defined networks are a huge benefit to OpenShift Virtualization users who want to create their own tenant networks either inside their own namespaces, or across namespaces with cluster UDNs. They offer cluster administrators the ability to organize network traffic by tenant instead of by cluster or namespace, and they offer users the security and ease-of-use of having their own isolated network spaces.

In this article, you have learned how to deploy user defined networks to allow your VMs to communicate on isolated tenant networks. To learn more about OpenShift Virtualization's features, head over to our OpenShift Virtualization landing page and explore how Red Hat’s products can benefit your organization.