Creating a Web Application Firewall in Red Hat OpenShift

In the last few years, several Red Hat customers have asked how to add a Web Application Firewall (WAF) to the OpenShift ingress to protect all externally facing applications.

A WAF is a Layer 7 capability that protects applications against some types of web-based attacks, including but not limited to Cross Site Request Forgery (CRSF),  Cross-Site Scripting (XSS) and SQL injection (for a more comprehensive list of all known web based attacks, see here).Unfortunately, OpenShift does not have these capabilities included within the default ingress router, and as a result, alternate solutions must be pursued.

But now, with the advent of the Gateway API and a new ingress stack for OpenShift, things are changing. In this article, we will demonstrate how you can protect your applications with a WAF.

OpenShift, Gateway API and WASM

Note: As of this writing, the OpenShift Gateway API implementation is in a dev preview support status and the functionality may change in the future, including the underlying technologies.

OpenShift, as a product, will adopt the new Gateway API standard. This feature is in dev preview behind a feature gate and can be activated as described in this article.

For the purpose of this article, we do not need to analyze all of the new features of the Gateway API (a good summary for that can be found here). However, the feature that we are most concerned with is that the reverse proxy implementation of the Gateway API for OpenShift is Envoy. In fact, by enabling the Gateway API feature, a clusterwide OpenShift Service Mesh is currently installed automatically. 

By having the power of the Istio at our disposal, we can easily inject new behavior in Envoy using WebAssembly (WASM) packaged plugins.WASM is a technology that compiles code such that it runs safely in a lightweight stack-based virtual machine.WASM was originally created to inject behavior in the browser to support sophisticated websites, but it turned out to be ideal for all kinds of plugin systems.To be compatible with Envoy, a WASM program must implement the proxy-wasm Application Binary Interface (ABI). This interface specification ensures that Envoy understands how to call the WASM program.

ModSecurity, OWASP WAF core rule set and Coraza

ModSecurity is a mature and stable Apache module that adds WAF capabilities to an Apache instance. ModSecurity has been adopted over time by other http servers and proxies products, such as ngnix via an appropriate connector.ModSecurity has recently been donated to Open Web Application Security Project (OWASP) and under this organization, will be maintained as an open source project with open and transparent governance.

One of the features of ModSecurity is that it can be configured to check for a set of rules via a Domain Specific Language (DSL) called SecLang. The OWASP organization maintains a default set of rules that cover the most common attack types. This set of rules is referred to as the OWASP core rule set.When configuring a WAF based on ModSecurity, it is a good practice to start with the OWASP core rule set and then tweak the ruleset based on one’s needs.

Coraza is a port of ModSecurity to the golang language (ModSecurity is originally written in C) with the objective of making ModSecurity more adopted by enterprises. Coraza Proxy WASM is an adaptation of Coraza to the proxy-wasm-ABI.

That being said, by leveraging these solutions, it should start to be clear how we are going to enhance OpenShift ingress with a WAF.

Putting it all together

The following diagram depicts the approach for injecting Coraza in the Envoy proxy which manages the ingress traffic:

In the diagram above, we can see in yellow, the CRs that we created, and in green, pods or groups of pods that are running. You can find this implementation in a declarative GitOps format within this repository.

At the bottom of the diagram, we can find the GatewayClass CR. When configured appropriately, it triggers the deployment of an ServiceMesh (Istio) control plane. Moving up higher in the diagram, we can see the Gateway CR. This manifest creates an Istio ingress gateway pod. When the Gateway CR is deployed the OpenShift Ingress Operator creates a wildcard DNS entry for this ingress endpoint. Additionally, we created a wildcard certificate using cert-manager to enable TLS for all the subdomains managed by this endpoint.

The last piece of configuration that the administrator needs to do is to inject the Coraza WAF code as a WASM plugin. This can be achieved with a WasmPlugin CR. Notice that this manifest references an OCI artifact containing the WASM code.

On the tenant side, we can deploy a simple application that demonstrates the use of a WAF now that the capability has been enabled. The example application can be found here. The key component in this application is the HttpRoute. By specifying the parentRefs section, we request the newly created Gateway to serve this route.

Now that we have enabled the Gateway API within OpenShift and configured the cluster to support our use case, and deployed the sample application, we can finally test and validate the functionality.

Let’s first get the current cluster base domain:

export cluster_base_domain=$(oc get ingress.config.openshift.io cluster --template={{.spec.domain}} | sed -e "s/^apps.//")

Then we can issue a request:

curl -I https://nginx-echo-headers.gwapi.${cluster_base_domain}
HTTP/2 200 
server: istio-envoy
date: Fri, 10 May 2024 18:23:50 GMT
content-type: text/plain
x-envoy-upstream-service-time: 2

Now, let’s make a request that looks a little more suspicious:

curl -I "https://nginx-echo-headers.gwapi.${cluster_base_domain}/?arg=<script>alert(0)</script>"
HTTP/2 403 
date: Fri, 10 May 2024 18:25:21 GMT
server: istio-envoy

The request was denied with a 403 error.

Within the Istio ingress gateway pod log, messages similar to the following will be displayed:

024-05-10T18:25:21.318531Z critical envoy wasm wasm log openshift-ingress.coraza-waf: [client "100.64.0.5"] Coraza: Warning. XSS Attack Detected via libinjection [file "@owasp_crs/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "7663"] [id "941100"] [rev ""] [msg "XSS Attack Detected via libinjection"] [data "Matched Data: XSS data found within ARGS_GET:arg: <script>alert(0)</script>"] [severity "critical"] [ver "OWASP_CRS/4.0.0-rc2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/242"] [hostname "10.129.2.11"] [uri "/?arg=<script>alert(0)</script>"] [unique_id "cmQXIPQegrompxpYJnD"]
126
…

Load test

By now, we know that the OWASP core rule set is being applied by our WAF./ But, was there a performance hit against the application? Let’s try to measure the delta with and without Coraza enabled.

To accomplish this task, we need to deploy another gateway without the plugin.

To generate load, we are going to use cassowary, a cross platform load testing tool. Here is our baseline, without Coraza:

podman run rogerw/cassowary:v0.14.1 -u https://nginx-echo-headers.gwapi-no-coraza.${cluster_base_domain} -c 10 -n 1500 -d 3
0 --disable-keep-alive
Starting Load Test with 1500 requests using 10 concurrent users
99% |███████████████████████████████████████ | [29s:0s]            30.056327229s

TCP Connect.....................: Avg/mean=17.74ms     Median=17.00ms  p(95)=22.00ms
Server Processing...............: Avg/mean=19.43ms     Median=19.00ms  p(95)=25.00ms
Content Transfer................: Avg/mean=0.00ms      Median=0.00ms   p(95)=0.00ms
Summary: 
Total Req.......................: 1500
Failed Req......................: 0
DNS Lookup......................: 1.00ms
Req/s...........................: 49.91

Now, let’s see how the same load performs with Coraza enabled:

podman run rogerw/cassowary:v0.14.1 -u https://nginx-echo-headers.gwapi.${cluster_base_domain} -c 10 -n 1500 -d 30 --disable-keep-alive
Starting Load Test with 1500 requests using 10 concurrent users
99% |███████████████████████████████████████ | [29s:0s]            30.075051016s

TCP Connect.....................: Avg/mean=18.14ms     Median=18.00ms  p(95)=23.00ms
Server Processing...............: Avg/mean=33.46ms     Median=33.00ms  p(95)=40.00msEnvoy
Content Transfer................: Avg/mean=0.00ms      Median=0.00ms   p(95)=0.00ms
Summary: 
Total Req.......................: 1500
Failed Req......................: 0
DNS Lookup......................: 1.00ms
Req/s...........................: 49.88

We can see that for this simple use case, the latency introduced by the WAF was around 14ms (33ms-19ms).

Also, we noticed that on the resource consumption side, the ingress gateway pod with Coraza (top in the picture below) consumes approximately one order of magnitude (x10) more memory and CPU.

Conclusions

In this article, we demonstrated an approach to add a WAF to our OpenShift ingress stack using the new Gateway API and injection via a WASM plugin. We used Coraza as the WAF implementation and the OWASP core rule set as our WAF policies. When attempting to deploy these assets in your own environment, please keep in mind that the Gateway API is still in dev preview in OpenShift and that the Coraza WASM plugin is not currently supported by Red Hat.

In addition to WAF and the core rule set, OWASP recommends sanitizing the HTTP headers following the OWASP Secure Headers Project “best practices”. These best practices can be tested at develop time, but one can also decide to sanitize the headers on the fly at run time. We built a custom WASM plugin to demonstrate this use case. The plugin can be found in this repository. This sample plugin attempts to satisfy most recommendations from the published HTTP Security Response Headers Cheat Sheet. 

In our opinion, WASM is a promising approach for enhancing the behavior of the Envoy reverse proxy especially when Envoy is controlled by Istio (multiple plugins can be attached to the same Envoy instance). We also recommend investigating project Kuadrant in detail which uses WASM to add authz/authn and rate-limiting capabilities (and more) to an Envoy proxy.