With the 1.0 release, Istio is preparing itself for prime time by companies developing container-native applications with a desire for adopting a service mesh solution.

However, there is a potential issue that may slow down the adoption rate: Pods that belong to the Istio mesh require elevated privileges in order to function properly.

In this article we will present the istio-pod-network-controller, a solution to mitigate this issue.

The Problem

As part of normal operations of the service mesh, Istio needs to manipulate the iptables rules of the pod in order to intercept connections to the application containers and inject all the behaviors that make Istio so great. Iptables rules are network-namespaced, so the changes made within a pod do not affect the other pods or the node on which the pod is running.

An init container is part of a Istio pod and is responsible for adding these iptables rules prior to starting the primary set of containers. Manipulating iptables rules is an action that requires elevated access through the NET_ADMIN capability. NET_ADMIN is a kernel capability that (as one can imagine) allows you to reconfigure the network of a linux machine. This means that a pod with that capability can not only add itself to the Istio mesh, but also start interfering with the networking configuration of the other pods along with the node itself. In general, it is not advisable to have application pods with that capability in a shared tenant cluster.

OpenShift provides a way to control the permissions (and in this case kernel capabilities) a pod can have through a mechanism called Security Context Context (SCC). There are a few out-of-the-box SCC profiles in OpenShift and an administrator can add more of them. The only-out-of-the box SCC profile that allows for the proper operation of Istio is the privileged profile. In order for pods within a namespace to be added to the Istio service mesh, the following command must be executed to gain access to the privileged SCC:

<span>oc adm policy add-scc-to-user privileged -z default -n </span><span>&lt;</span><span>target-namespace</span><span>&gt;</span>

Essentially, root access has been given to all the pods in this namespace and is not typically advisable due to the security implications when running normal applications.

While this issue has been on the minds of those in the Istio community, Kubernetes up until recently did not have a mechanism to control the permissions given to a pod. Starting in Kubernetes 1.11, the Pod Security Policy (PSP)  feature has been introduced as a beta feature. PSPs function in a similar fashion as SCCs. As soon as other Kubernetes distributions start to support out-of-the-box PSPs, it will become apparent that pods in the Istio mesh need (too) elevated privileges to run properly.

The Solution

One method for mitigating this issue is to move the logic that configures the iptables rules of the pod out of the pod itself.

The result is a DaemonSet controller, called the istio-pod-network-controller, that monitors for the creation of new pods and configures the appropriate iptables rules in these new pods as soon as they created. The diagram below depicts the overall architecture of the solution:

The flow is the following:

1. A new pod is created
2. The istio-pod-network-controller deployed on the node where the pod is created determines whether the new pod should belong to the Istio mesh and therefore must be initialized.
3. If yes, the istio-pod-network-controller initializes the iptables rules of the new pod and marks the pod as initialized via an annotation.
4. An init container within the pod waits for the initialized annotation to be present ensuring that the application container and the sidecar Envoy proxy start only after the iptables initialization is complete
5. The sidecar container and the application container start.

With this solution in place, the pods running in the Istio mesh only need access to the nonroot SCC. The nonroot SCC is required due to the fact the Envoy sidecar needs to run with a specific non-root user id.

Ideally, it would be desired that Istio applications to run with the restricted SCC, the default within OpenShift. Though, the nonroot SCC is only a marginal relaxation over restricted and in general an acceptable compromise. Regardless, it is a vast improvement compared to requiring each Istio application pod run using the privileged SCC.

Instead, it is the istio-pod-network-controller that utilizes the privileged profile and NET_ADMIN capability to be able to modify other pod’s iptables rules. This is generally an acceptable solution as this component would be installed and managed by cluster administrators in a similar fashion to the Istio control plane.

Installation Instructions

With the assumption that Istio has been successfully installed in the istio-system namespace per the installation guide, clone the repo and then the following command can be executed to install the istio-pod-network-controller using Helm:

<span>helm template -n istio-pod-network-controller --set kubernetesDistribution=OpenShift ./chart/istio-pod-network-controller | oc apply -f -</span>

For additional deployment scenarios, please refer to the instructions at the istio-pod-network-controller GitHub repository

In the same repo you can also find instructions to test the istio-pod-network-controller has been deployed correctly.

Conclusion

The istio-pod-network-controller is an option to make Istio deployments more secure. It does so by removing the need to run pods in the Istio mesh with the privileged SCC and allows them to run just with the nonroot SCC. If you decide to adopt this solution, please keep in mind that this is currently a best effort project and not officially supported by Red Hat.