Demystifying Multus

Multus is the open source project that enables Kubernetes pods to attach to multiple networks. It does this by acting as a ‘meta’ plug-in, which is a fancy name for a plug-in that calls other plug-ins. First, let’s take a quick look at the ‘pluggable’ nature of container networking.

In 2017, the Cloud Native Computing Foundation (CNCF) voted to accept the Container Networking Interface (CNI) as the 10th hosted project. The CNI project is a minimal specification to define a common interface between the network plug-ins and container execution. CNI has 3 main components:

• CNI specification – defines the API between runtimes and network plug-ins
• Plug-ins – provide network setup for different use cases (reference examples)
• Library – a Go implementation of the CNI specification that runtimes can use

Currently, the default OpenShift CNI is OpenShift SDN (network-policy), which configures an overlay network using Open vSwitch (OVS 2.11). The following diagram shows the CNI options for OpenShift and the status of each (supported, validated, etc. …):

In OCP 3.x by default, a standard pod configuration allows for a connection to one network interface.

Multus allows pods to have multiple network interface connections that can address various use cases; for example:

• Splitting the control/data plane traffic, mostly applicable to vnf/cnf applications
• Legacy/storage intensive applications
• Multitenant networks

OpenShift provides the following CNI plug-ins for creating additional networks in your cluster:

• Bridge: allows Pods on the same host to communicate with each other and the host
• Host-device: allows Pods access to a physical Ethernet network device on the host system
• Macvlan: allows Pods on a host to communicate with other hosts and Pods on those hosts by using a physical network interface. Each Pod that is attached to a macvlan-based additional network is provided a unique MAC address
• Ipvlan: allows Pods on a host to communicate with other hosts and Pods on those hosts, similar to a macvlan-based additional network. Unlike a macvlan-based additional network, each Pod shares the same MAC address as the parent physical network interface
• SR-IOV: allows Pods to attach to a virtual function (VF) interface on SR-IOV capable hardware on the host system

Currently, setting up secondary network interfaces is a ‘day 2’ operation. The following is an example where I’ve added a bridge ‘additional network’ to the network operator (highlighted):

Add network-

oc edit networks.operator.openshift.io cluster
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: operator.openshift.io/v1
kind: Network
metadata:
  creationTimestamp: 2020-02-05T13:13:48Z
  generation: 2
  name: cluster
  resourceVersion: "24716"
  selfLink: /apis/operator.openshift.io/v1/networks/cluster
  uid: 3f48c7b7-4e1c-443e-8b52-d161e6e48adb
spec:
  additionalNetworks:
  - name: multus1
rawCNIConfig: '{ "cniVersion": "0.3.1", "type": "bridge", "master": "eth1", "ipam":
   { "type": "static", "addresses": [ { "address": "191.168.1.1/24" } ] } }'
type: Raw
  clusterNetwork:
  - cidr: 10.128.0.0/14
hostPrefix: 23
  defaultNetwork:
type: OpenShiftSDN
  logLevel: ""
  serviceNetwork:
  - 172.30.0.0/16

Then create a Pod with the following annotation (note the name -multus1 defined above):

apiVersion: v1
kind: Pod
metadata:
  name: example
  labels:
app: hello-openshift
  namespace: default
  annotations:
k8s.v1.cni.cncf.io/networks: multus1
spec:
  containers:
- name: hello-openshift
   image: openshift/hello-openshift
   ports:
     - containerPort: 8080

And when the Pod is spun up, you can see the default plug-in ‘eth0’ and the secondary network interface ‘net1’;

kind: Pod
apiVersion: v1
metadata:
  name: multus-example
  namespace: default
  selfLink: /api/v1/namespaces/default/pods/multus-example
  uid: 0d68eb92-d74c-4f08-8924-6dcdcde26e7a
  resourceVersion: '113663'
  creationTimestamp: '2020-02-05T19:38:15Z'
  labels:
app: hello-openshift
  annotations:
k8s.v1.cni.cncf.io/networks: multus1
k8s.v1.cni.cncf.io/networks-status: |-
   [{
       "name": "openshift-sdn",
       "interface": "eth0",
       "ips": [
           "10.129.2.19"
       ],
       "dns": {},
       "default-route": [
           "10.129.2.1"
       ]
   },{
       "name": "multus1",
       "interface": "net1",
       "ips": [
           "191.168.1.1"
       ],
       "mac": "fa:70:12:98:e4:5f",
       "dns": {}
   }]

For high performance networking workloads SR-IOV provides near-native networking performance on bare-metal.This feature is Tech Preview for OpenShift 4.3.

The SR-IOV operator can be installed from the console and provides the following features:

• Discover the SR-IOV network device in the cluster.
• Initialize the supported SR-IOV NIC models on nodes.
• Provision the SR-IOV network device plug-in on nodes.
• Provision the SR-IOV CNI plug-in executable on nodes.
• Provision the Network Resources Injector in the cluster.
• Manage configuration of SR-IOV network device plug-in.
• Generate NetworkAttachmentDefinition custom resources (CR) for the SR-IOV CNI plug-in.

Tested CNI plug-ins can be found here.