Network Policy Objects in Action

Use Network Policy Objects to restrict traffic flow between application components or microservices with a demonstration from this video and post. Network Policy Objects allow you to define a policy to determine when traffic is allowed to flow to specific services and prevent traffic to other services.

As of the writing of this article, this feature is in "Technology Preview" in OpenShift 3.6.

Prerequisites

To test this concept, you have to install an OpenShift cluster with redhat/openshift-ovs-networkpolicy using os_sdn_network_plugin_name in the Ansible hosts file or enable network policy in an existing cluster as explained in https://docs.openshift.com/container-platform/3.6/admin_guide/managing_networking.html#admin-guide-networking-networkpolicy

Understanding the Sample Application

To look at a practical example that showcases Network Policy Objects, we'll use a sample application that is made up of a group of microservices from this Git repository: https://github.com/VeerMuchandi/microservices-on-openshift.

Deploying these microservices will provide you a use case for understanding how Network Policy Objects are useful.

This application has four microservices written using different technologies just to showcase how polyglot microservices talk to each other. Some of these microservices have application logic and a database. All the components run as containers on OpenShift.

1. Front-end Service that provides a UI to register users, allows you to login, and displays the tweets for all the registered users, written in PHP.
2. Email Service that can send emails to a registered user, written in Python, that saves data in MySQLdb.
3. Twitter Service that pulls tweets for a registered user, written in Java running on Tomcat. This service is called by the Front-end HTML directly.
4. User Registration Backend that registers the users, saves the registered user in a MongoDB, and is written in Node.Js. This service is called by the Front-end HTML and does the job of user registration. Once the user is registered, it calls the Email service to send an email confirmation.

This is all explained in this application's Git repository. For our network policy use case, we will deploy these microservices into three different projects:

1. msclient project hosts the front-end service.
2. msservices project hosts the user-registration back-end service and the Twitter service.
3. msinfra project hosts the email service.

The deployment model for these microservices and the expected interations (in order) are shown in the figure below. I recommend spending some time on this model to get a better understanding.

Deploy the application

Prerequisites

• For Email Service to work, you will need a Gmail account to send emails. You will also have to turn on access for less secure apps on this Gmail account so that the application can log in and send emails. Caution: Don't use your personal Gmail. Create a new one that you don't use so that you don't care if it is hacked.
• For Twitter Service to work, you will need access to keys and tokens to work with Twitter. Refer to the documentation at https://dev.twitter.com/oauth/overview/application-owner-access-tokens for more information.

Step 1:  Clone the repository to your local machine:

git clone https://github.com/VeerMuchandi/microservices-on-openshift

and change to the microservices-on-openshift/installscripts folder:

cd microservices-on-openshift/install-scripts

Step 2: Create three projects (after you have already logged into the OpenShift cluster):

oc new-project msclient

oc new-project msservices

oc new-project msinfra

Step 3: Set environment variables

Edit 1.setVariable.sh script to set the following values:

• OSE_DOMAIN to your OpenShift domain name
• OSE_CLIENT_PROJECT to msclient
• OSE_SERVICES_PROJECT to msservices
• OSE_INFRA_PROJECT to msinfra
• FROM_GMAIL to your Gmail account from where you want to send Gmails
• FROM_GMAIL_PASSWORD password for the above Gmail account
• And values for TWITTER_CONSUMER_KEY, TWITTER_CONSUMER_SECRET<, TWITTER_OAUTH_ACCESS_TOKEN, and TWITTER_OAUTH_ACCESS_TOKEN_SECRET

Run the script as follows

source 1.setVariable.sh

Step 4: Deploy microservices by running the following scripts:

This script will build and deploy Email Service and the MySQLdb in the msinfra project.

source 2.deployEmailSvc-PythonMySQL.sh

This script will build and deploy Twitter Service in the msservices project:

source 3.deployTwitter-Tomcat.sh

This script will build and deploy User Registration Back-end Service in the msservices project

source 4.deployUserRegBackend-NodejsMongo.sh

This script will build and deploy Front-end Service in the msclient project:

source 5.deployFrontend-PHP.sh

In a few minutes, you will notice all the services up and running. You should see the following:

$ oc get pods -n msclient

NAME              READY     STATUS      RESTARTS   AGE

userreg-1-build   0/1       Completed   0          1m

userreg-1-tm3x8   1/1       Running     0          1m
$ oc get pods -n msservices

NAME                  READY     STATUS      RESTARTS   AGE

mongodb-1-2zx66       1/1       Running     0          2m

twitter-api-1-build   0/1       Completed   0          2m

twitter-api-1-srvvm   1/1       Running     0          1m

userregsvc-1-b4xq3    1/1       Running     0          1m

userregsvc-1-build    0/1       Completed   0          2m
$ oc get pods -n msinfra

NAME               READY     STATUS      RESTARTS   AGE

emailsvc-1-build   0/1       Completed   0          3m

emailsvc-1-grb00   1/1       Running     0          2m

mysql-2-ffj05      1/1       Running     0          3m
$ oc get route -n msclient

NAME      HOST/PORT                                   PATH      SERVICES   PORT       TERMINATION   WILDCARD

userreg   userreg-msclient.apps.devday.ocpcloud.com             userreg    8080-tcp                 None

Step 5: Test your application

Access the UI for the frontend and register a few users. In my case, the URL is http://userreg-msclient.apps.devday.ocpcloud.com. If email is configured correctly the user that is registered should receive an email. Also, you should be able to pull the tweets for the configured user.

So far, we've deployed a few microservices and used them on OpenShift.

Now we will see why we need Network Policy Objects

We have a hack in the Front-end Service code. If you access your <>/hack.php, it talks to the MySQLdb in the msinfra project and pulls all the data and shows it as below.

$ curl http://userreg-msclient.apps.devday.ocpcloud.com/hack.php

Here is the list of emails sent:

From Address: xxx@gmail.comTo Address: yyy@gmail.comSubject: New registrationWhen: 2017-08-22

From Address: xxx@gmail.comTo Address: zzz@gmail.comSubject: New registrationWhen: 2017-08-22

End of the list

Issue: Ideally UI should not reach a database behind a microservice, but instead should access it via the APIs exposed by the microservice. It is a well-known architecture practice that if you are running an application in a multi-tiered architecture, your UI layer should not be talking to th database directly but go via APIs exposed by business logic. In fact, the database is generally firewalled, and only ports from the services that are supposed to call the database are allowed. If you are using a flat network such as ovs-subnet, these fine-grained network controls between services are not possible.

This post explains what Network Policy Objects are and how they can be used to control network traffic in theory: https://blog.openshift.com/whats-new-in-openshift-3-5-network-policy-tech-preview/.

Step 6: Network Policy Objects to the rescue

Let us now add Network Policy controls to our microservices so that the above kind of hack is not allowed. We'll add policies that allow specific traffic to your services.

Let's protect the MySQLdb behind the Email Service so that it is only accessible from the Email Service. Create a file with the following content:

$ cat allow-3306.yaml

kind: NetworkPolicy

apiVersion: extensions/v1beta1

metadata:

  name: allow-3306

spec:

 podSelector:

   matchLabels:

     app: mysql

 ingress:

 - from:

   - podSelector:

       matchLabels:

         app: emailsvc

   ports:

   - protocol: TCP

     port: 3306

Here we are allowing traffic coming from pods that have labels matching app: emailsvc to reach the MySQL pod that has label app:mysql. Look at the labels assigned to pods and services to understand how this works.

$ oc get svc -n msinfra --show-labels

NAME       CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE       LABELS

emailsvc   172.30.42.184           8080/TCP   1h        app=emailsvc,microservice=emailsvc

mysql      172.30.43.224           3306/TCP   1h        app=mysql,microservice=emailsvc
$ oc get pods -n msinfra --show-labels | grep Running

emailsvc-1-grb00   1/1       Running     0          1h        app=emailsvc,deployment=emailsvc-1,deploymentconfig=emailsvc,microservice=emailsvc

mysql-2-ffj05      1/1       Running     0          1h        app=mysql,deployment=mysql-2,deploymentconfig=mysql,microservice=emailsvc

Now create this policy in the msinfra project as follows:

$ oc create -f allow-3306.yaml -n msinfra

networkpolicy "allow-3306" created

As soon as the policy is added, if you test the hack again, it should fail with 504 Gateway Time-out as shown below:

$ curl http://userreg-msclient.apps.devday.ocpcloud.com/hack.php

504 Gateway Time-out

The server didn't respond in time.

Step 7: Fully protect your microservices

Note You can clone this Git repository to download all these policy files: https://github.com/VeerMuchandi/usingnetworkpolicyobjects

Let's add more policies now.

Completely deny all traffic to msclient project:

$ cat default-deny.yaml

kind: NetworkPolicy

apiVersion: extensions/v1beta1

metadata:

name: default-deny

spec:

podSelector:

Create this policy:

$ oc create -f default-deny.yaml -n msclient

networkpolicy "default-deny" created

Now if you test your UI, it should say Application is not available.

You can see that the route (from loadbalancer) is reaching the service at port 8080:

$ oc get route -n msclient --show-labels

NAME      HOST/PORT                                   PATH      SERVICES   PORT       TERMINATION   WILDCARD   LABELS

userreg   userreg-msclient.apps.devday.ocpcloud.com             userreg    8080-tcp                 None       app=userreg,microservice=userreg

So let's allow port 8080 for the the userreg service to enable communication with UI as below:

$ cat allow-8080-frontend.yaml

kind: NetworkPolicy

apiVersion: extensions/v1beta1

metadata:

  name: allow-8080-frontend

spec:

  podSelector:

    matchLabels:

      app: userreg

  ingress:

  - ports:

    - protocol: TCP

      port: 8080
$ oc create -f allow-8080-frontend.yaml -n msclient

networkpolicy "allow-8080-frontend" created

Now your front-end URL should work again!

Next, Default Deny all traffic to msservices project:

$ oc create -f default-deny.yaml -n msservices

networkpolicy "default-deny" created

Now allow 8080 to User Registration Backend and Twitter Services:

$ cat allow-8080-userregsvc.yaml

kind: NetworkPolicy

apiVersion: extensions/v1beta1

metadata:

  name: allow-8080-userregsvc

spec:

  podSelector:

    matchLabels:

      app: userregsvc

  ingress:

  - ports:

    - protocol: TCP

      port: 8080
$ cat allow-8080-twitter.yaml

kind: NetworkPolicy

apiVersion: extensions/v1beta1

metadata:

  name: allow-8080-twitter

spec:

  podSelector:

    matchLabels:

      app: twitter-api

  ingress:

  - ports:

    - protocol: TCP

      port: 8080
$ oc create -f allow-8080-userregsvc.yaml -n msservices

networkpolicy "allow-8080-userregsvc" created

$ oc create -f allow-8080-twitter.yaml -n msservices

networkpolicy "allow-8080-twitter" created

Allow MongoDB to be accessed only by User Registration Service:

$ cat allow-27017.yaml

kind: NetworkPolicy

apiVersion: extensions/v1beta1

metadata:

  name: allow-27107

spec:

 podSelector:

   matchLabels:

     app: mongodb

 ingress:

 - from:

   - podSelector:

       matchLabels:

         app: userregsvc

   ports:

   - protocol: TCP

     port: 27017
$ oc create -f allow-27017.yaml -n msservices

networkpolicy "allow-27107" created

Deny all traffic to msinfra project:

$ oc create -f default-deny.yaml -n msinfra

networkpolicy "default-deny" created

We already have only the Email Service accessing the MySQldb. We will now have to allow connection to the Email Service in the msinfra project from the User Registration Service in a msservices project. While you can select the destination in this case, you cannot select a specific pod on the source side when the namespaces are different. So we will have to open up the namespace msservices to be able to reach the Email Service in msinfra.

First, we will need a label to identify the (traffic) source project msservices. Your cluster-admin may have to do this.

# oc label namespace msservices project=userregservices

namespace "msservices" labeled

Now let's add the policy that allows services in msservices project to talk to the Email Service in the msinfra project. Note the namespaceSelector here, as we are opening traffic flow from msservices namespace here.

$ cat allow-8080-emailsvc.yml

kind: NetworkPolicy

apiVersion: extensions/v1beta1

metadata:

  name: allow-8080-emailsvc

spec:

  podSelector:

    matchLabels:

      app: emailsvc

  ingress:

  - from:

    - namespaceSelector:

        matchLabels:

          project: userregservices

    ports:

    - protocol: TCP

      port: 8080
$ oc create -f allow-8080-emailsvc.yml -n msinfra

networkpolicy "allow-8080-emailsvc" created

Now we are all set. We have policies in place that prevent services crossing boundaries. Test the application to make sure the calls between microservices are all going through.

Summary

You should now understand:

• Why Network Policy Objects are important.

• How to create policies that deny all the traffic.

• How to open up traffic to specific destinations based on labels.

• How to open up traffic that crosses across projects.

Acknowledgements

Dan Winship helped me at every stage of this testing Network Policy Objects and got me up to speed on using this feature. Many thanks to Dan.