Performing large-scale network testing on Red Hat OpenShift: A 100 Gbps approach

This article aims to benchmark the results for 100 Gbps network interface assets and develop tools for driving the load.

Recently, a customer faced a network test issue in which the anticipated bandwidth was observed at the node level, but there was a reduction of 25% to 30% at the pod level. The customer could resolve the issue by using HostNetwork for the pods because it bypasses OVN-K hops. However, this approach can have security implications, so that's not the ideal solution.

The analysis further revealed that the customer application exhibited low latency and high packets per second (PPS), especially when routing traffic directly through the host network interface. The customer's environment had bare metal worker nodes, each equipped with two 100GB NICs. I tested the network with iPerf3 and validated the network speeds between pods. No automated procedure to conduct these tests existed. The customer is actively seeking a script to validate the scenario.

Get started

I conducted a large-scale network test for Red Hat OpenShift clusters hosted on bare metal Linux worker nodes. I chose iPerf3 and k8s-netperf to stress the network component by saturating the 100Gbps network.

k8s-NetPerf

This is a client-server model with two executables: netperf and netserver. You can execute netperf on the remote system's inetd or as a standalone daemon. netperf establishes a "control connection" to the remote system when executed, which passes test configuration information and results to and from the remote system. Regardless of the test type, the control connection is TCP-based and implemented by BSD sockets. The control connection can use IPv4 or IPv6. Once the control connection is active and you supply the configuration information, it establishes a separate "data" connection for the measurement using the API and protocol appropriate for the test.

The data connection is terminated upon completion. The results from the netserver are passed back through the control connection and combined with netperf results for display. netperf places no traffic on the control connection while a test is in progress.

Specific TCP options, such as SO KEEPALIVE, may put packets out on the control connection while a test is in progress. Generally speaking, this does not affect the results.

I use k8s-netperf to perform the following tests:

• Test end-to-end latency (round-trip times or RTT).
• Establish one thread per stream.

iPerf3

iPerf is a tool for actively measuring the maximum achievable bandwidth on IP networks. It supports tuning parameters related to timing, protocols, and buffers.

The client node generates traffic towards the server node. iPerf measures network throughput and displays an estimate of network speed between the client and server. For each test, it reports the measured throughput and bitrate, loss, and other parameters.

I used the multithreaded beta release of iPerf3 in my tests. iPerf3 is a new implementation with the goal of a smaller, simpler code base and a library version of the functionality you can use in other programs. It's useful in the following cases:

• Tests packet loss and delay jitter.
• Troubleshooting network performance.
• Multiple threads per process.

Choosing one or the other tool depends on the use case.

Testbed topology

The infrastructure consisted of three Master Nodes and eight Worker Nodes. The Master and Compute Nodes were Dell PowerEdge R650 Servers, each equipped with a 100Gb/S physical interface backed by a Mellanox network controller. Here is the server topology in the OpenShift diagram.

Baremetal Server topology in OpenShift deployment

Hardware stack

I performed the test on internal lab hardware consisting of 12 x Dell R650 servers. Each server is equipped with the following:

• CPU: 28 Physical cores - Dual Socket (56 Threads)
  • Model name: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz
• Physical network interface: 100Gb/s
• Ethernet controller: Mellanox Technologies MT28908 Family [ConnectX-6]
• Disk: Micron Technology Inc 7450 PRO NVMe SSD
  • SATA-attached SSD: C620 Series Chipset Family SATA Controller
• Memory: DDR4, 512GB

Software stack

The software stack includes the following components:

• OpenShift: 4.13.17
  • oc-client: 4.13.17
• k8s-netperf: v0.1.16
  • Go Version: 1.19

Build and execution

This section describes how to build the k8s-netperf locally for an alternate architecture.

$ git clone http://github.com/cloud-bulldozer/k8s-netperf -b v0.1.16
$ cd k8s-netperf/
$ make build

k8s-netperf configuration

This section captures the configuration leveraged during k8s-netperf testing. This configuration will execute five samples. Each sample has a duration of 60 seconds with a payload size of 16KB.

---
tests:
 - TCPStream:
   parallelism: 30
   profile: "TCP_STREAM"
   duration: 60
   samples: 5 
   messagesize: 16384

I cloned e2e-benchmarking to trigger the network testing. You need a KUBECONFIG pointing to the cluster you want to run k8s-netperf against. The custom configuration used allows user to different combinations 

$ 
$ git clone https://github.com/cloud-bulldozer/e2e-benchmarking
$ cd e2e-benchmarking/workloads/network-perf-v2/
$ export WORKLOAD=full-run.yaml  <--- Points to our custom configuration
$ ./run.sh

The above command will run the workload and generate output.

You can get the scripts from this GitHub repo.

Results analysis

Using netperf and iPerf3 with 25 clients demonstrated the capability to drive network tests reaching ~95Gbps with a 1500 MTU and 16384k msg_size without system tuning, indicating confidence in the network tools employed and allowing users to drive large-scale network testing.

Output:

Pod2Pod network perf test result:

Host network perf test results:

The ServerSide CPU usage peaked at 72% for 25 streams, whereas the ClientSide experienced a CPU usage that reached 108%.

The CPU Usage for the default (one) stream peaked on the ServerSide at 18%, while on the ClientSide, it reached 45%.

Wrap up

The large-scale network testing focusing on a 100 Gbps approach on OpenShift provided valuable insights into the system performance. The analysis revealed a reduction in bandwidth at the pod level, prompting exploration of solutions such as using HostNetwork for pods, albeit with associated security considerations.

The evaluation incorporated tools like k8s-netperf and iPerf3, leveraging a robust infrastructure of Dell PowerEdge R650 servers equipped with 100Gb/s interfaces and backed by Mellanox network controllers. The tests, executed with careful consideration of hardware and software stacks, demonstrated the ability to drive network tests at approximately 95Gbps, showcasing the effectiveness of the network tools employed, including k8s-netperf (Netperf and iPerf3).