A critical security vulnerability in Kubernetes (CVE-2018-1002105) affecting OpenShift Container Platform 3 was patched by Red Hat in December 2018. The OpenShift API Server is a component which handles all the API requests for OpenShift Container Platform 3. The API Server had a vulnerability in the ‘upgradeawarehandler’ type which acts as a reverse proxy to the compute nodes and other services added as API Extensions. A flaw in the reverse proxy prevented it from closing connections from the downstream service when there was an error. This allowed the user making the API call to escalate their privileges to those of the API Server. For details of the patch, see the upstream commit for Kubernetes.
Context
The OpenShift API Server is included in all OpenShift 3 installations and handles all the administration tasks for the cluster. Administrators and developers don’t usually call the API directly, but use the ‘oc’ binary, which calls the API server. Further information including detection, mitigation, and patching instructions can be found in the vulnerability article.
Amongst other things, the API server provides the functionality for the following oc commands:
-
oc exec
-
oc port-forward
-
oc rsh
It does this by acting as a reverse proxy to the kubelet running on the compute nodes. When connecting to the kubelet in order to fulfill any of the above commands it opens a websocket connection which connects stdin, stdout, or stderr to the administrator or developers original call. For more information, read the Remote Commands section of the OpenShift Container Platform 3.11 architecture guide.
The API Server also acts as a reverse proxy when implementing the API Aggregation feature of Kubernetes. API aggregation enables the installation of additional Application Programming Interfaces (APIs) into the core API Server. Those additional APIs are referred to as API extensions by upstream Kubernetes. API Extensions allow an architect to extend the features of OpenShift Container Platform 3.
Exploiting pod attach/exec/port-forward
In order to use pod attach/exec/port-forward permissions to exploit this vulnerability in OpenShift Container Platform 3, an attacker would need to have the permission granted by the ‘admin’, or ‘edit’ OpenShift roles normally allocated to developers.
When connecting to the kubelet, the API Server uses certificate based authentication. The certificate used to connect to the kubelet grants it ‘cluster-admin’ privileges, which allows full access to the kubelet. Authorization of the actions allowed by an attacker are only checked when the request is received by the API Server. If an attacker sends a request which is authorized, such as exec to their own pod, it is passed onto the kubelet.
However, if an attacker sends a request which includes a Connection Upgrade HTTP Header and is denied, the kubelet returns a non HTTP 101 status response, but the connection between an attacker and the kubelet is not closed. This results in the attacker being able to send subsequent requests to the kubelet without the API Server checking if the attacker is authorized to perform those requests. The patch rectifies the situation by checking the response from the downstream service (kubelet in this case) and closes the connection made by a potential attacker to the API Server.
If there are any privileged containers running on the compute node alongside a pod which an attacker has permissions to run remote commands on, they can send a subsequent request to the kubelet via the API Server’s open websocket to the privileged container instead. For example in OpenShift 3.11 the ‘ovs’ container in the ‘openshift-sdn’ namespace runs as a privileged container on every node. Getting access to this container allows an attacker to remount the host filesystem.
Exploiting API Extensions
By default the Service Catalog API Extension is installed in OpenShift Container Platform from versions 3.7 to 3.11. It was a Tech Preview feature in 3.6 and was not installed by default. A user can check if they have it installed in the cluster by issuing an API request to the ‘servicecatalog.k8s.io’ endpoint. The following instructions show how to do that by using the ‘oc proxy’ command, which sets a reverse proxy to forward request from port 8001 on the loopback device locally to an API Server which you are currently logged in.
$ oc proxy Starting to serve on 127.0.0.1:8001
Then in a new terminal window:
$ curl http://localhost:8001/apis/servicecatalog.k8s.io/v1beta1/ { "kind": "APIResourceList", "apiVersion": "v1", "groupVersion": "servicecatalog.k8s.io/v1beta1", "resources": [ { "name": "clusterservicebrokers", "singularName": "", "namespaced": false, "kind": "ClusterServiceBroker", "verbs": [ "create", "delete", "deletecollection", "get", "list", "patch", "update", "watch" ] },...
While ‘oc proxy’ commands add authentication options to the API call forwarded to the downstream service, the authentication options are not required to access that endpoint. This can be tested by changing the hostname from the proxy server URL to that of the OpenShift web console.
While an unauthenticated user can send requests to this service catalog discovery endpoint, they should not be able to invoke any of the API calls listed as resources by the HTTP response to that call, for example, http://localhost:8001/apis/servicecatalog.k8s.io/v1beta1/clusterservice…
Using this vulnerability exposed by the API Server, an attacker is able to create a malicious request to the service catalog which returns an error from the downstream service catalog to the API Server. An attacker can then use the unclosed HTTP connection, to send a subsequent request directly to the service catalog, and can now add X-Remote-User and X-Remote-Group headers to fake any user or group, including cluster-admin.
The extension-apiserver-authentication-reader role allows an API Extension to take part in the TLS certificate authentication chain. That is, it will be able to read the extension-apiserver-authentication configmap in the kube-system namespace which contains a CA cert. Any client key signed by this CA is trusted for authentication via user and group HTTP headers. The API Server sends the authenticated user and groups via those headers and the API extension will know that these values are trustworthy and will use them to do delegated authorization. Controlling these headers therefore allows arbitrary privilege escalation.
Knowing that the API extension needs the extension-apiserver-authentication-reader role for this means that we can query the cluster for service accounts which have a binding to that role.
$ oc get rolebindings -n kube-system -o=custom-columns=ROLE:.roleRef.name,USERNAME:.userNames | grep extension-apiserver-authentication-reader
If there is any service account with a role binding linking it to the extension-apiserver-authentication-reader role, that API extension could potentially be used to exploit this vulnerability. We have found the service catalog and metrics service API Extensions to be vulnerable, however your cluster may have another custom API extension making use of this aggregated API server feature. Also, we have found that Openshift template service broker is not affected, so it’s not necessary to disable it.
If we know a target API extension exists on the cluster, we can issue a malicious request via the API Server which will cause the downstream API extension to return an error to the API Server. However, as with the pod exec attack, the HTTP connection will remain open for the attack to send malicious requests directly to the API extension via an open websocket. Those requests include a client key, signed by the trusted CA cert, and the request will be allowed by the API extension. This allows an attacker to make any API calls to the downstream service. In the case of the service catalog, that allows an attacker to create a service broker in any namespace, and on any node.
Call to Action
If you are running an OpenShift Container Platform 3 cluster, apply the patches listed in the vulnerability article as soon as possible. If the patches cannot be applied immediately, consider applying both of the mitigations offered. If you trust the users of your cluster with pod attach/exec/port-forward permissions, it may only be necessary to disable any API extensions installed on your cluster until you can apply the full patch.
Thanks goes to the Kubernetes Security team for reporting this issue to Red Hat Product Security under embargo. Also, I would like to thank Red Hatters David Eads and Stefan Schimanski for their help writing this article, as well as Red Hat’s Paul Harvey and Sam Fowler for their help investigating the vulnerability and its impact.
저자 소개
Specializing in Kubernetes, container runtimes, and web applications, Jason Shepherd is a principal security engineer in Red Hat's Product Security team. With a passion for open source and dedication to client success, Shepherd is your go-to guy for security assessment and data for security audits.
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.