When explaining Kubernetes to people new in the space I noticed that the concept of services is often not well understood. To help you better understand what services are and how you can troubleshoot them, we will have a look at a concrete setup and discuss the inner workings of services in this post.
In a nutshell, Kubernetes services are an abstraction for pods, providing a stable, virtual IP (VIP) address. As pods may come and go, for example in the process of a rolling upgrade, services allow clients to reliably connect to the containers running in the pods, using the VIP. The virtual in VIP means it's not an actual IP address connected to a network interface but its purpose is purely to forward traffic to one or more pods. Keeping the mapping between the VIP and the pods up-to-date is the job of kube-proxy, a process that runs on every node, which queries the API server to learn about new services in the cluster.
Enough theory, let's have a look at a concrete example: in the following I'm using a simple containerized HTTP service that exposes endpoint0/
at port 9876
, returning a JSON payload echoing the server it is serving from along with some other infos.
Initially, let's deploy a pod under the supervision of a ReplicationController along with a Service, resulting in:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sise-bkvhs 1/1 Running 0 9m
$ kubectl describe pod sise-bkvhs
Name: sise-bkvhs
Namespace: namingthings
Security Policy: restricted
Node: 192.168.99.100/192.168.99.100
Start Time: Tue, 18 Apr 2017 09:45:22 +0100
Labels: app=sise
Status: Running
IP: 172.17.0.2
Controllers: ReplicationController/sise
...
You can now, from within the cluster, access the pod directly via its assigned IP 172.17.0.2
:
$ curl 172.17.0.2:9876/endpoint0
{"host": "172.17.0.2:9876", "version": "0.4.0", "result": "all is well"}
This is however, as mentioned above, not advisable since the IPs assigned to pods may change. Enter services:
$ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
simpleservice 172.30.40.155 80/TCP 1m
$ kubectl describe svc simpleservice
Name: simpleservice
Namespace: namingthings
Labels: name=simpleservice
Selector: app=sise
Type: ClusterIP
IP: 172.30.40.155
Port: 80/TCP
Endpoints: 172.17.0.2:9876
Session Affinity: None
No events.
From within the cluster we can now access simpleservice
like so:
$ curl 172.30.40.155:80/endpoint0
{"host": "172.30.40.155", "version": "0.4.0", "result": "all is well"}
So far so go. But what makes the VIP 172.30.40.155
forward the traffic to the pod? The answer is: IPTables. Think of IPTables as a long list of rules that tells the Linux kernel what to do with a certain IP package.
Looking at the rules that concern our service (executed on a cluster node) yields:
$ sudo iptables-save | grep simpleservice
-A KUBE-SEP-ASIN52LB5SMYF6KR -s 172.17.0.2/32 -m comment --comment "namingthings/simpleservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-ASIN52LB5SMYF6KR -p tcp -m comment --comment "namingthings/simpleservice:" -m tcp -j DNAT --to-destination 172.17.0.2:9876
-A KUBE-SERVICES -d 172.30.40.155/32 -p tcp -m comment --comment "namingthings/simpleservice: cluster IP" -m tcp --dport 80 -j KUBE-SVC-IKIIGXZ2IBFIBYI6
-A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -j KUBE-SEP-ASIN52LB5SMYF6KR
Above you can see the four rules that kube-proxy has thankfully added to the routing table, essentially stating that TCP traffic to 172.30.40.155:80
should be forwarded to 172.17.0.2:9876
, which is our pod.
Let's now add a second pod by scaling up the ReplicationController supervising it:
$ kubectl scale --replicas=2 rc/sise
replicationcontroller "sise" scaled
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
sise-bkvhs 1/1 Running 0 11m
sise-vncz4 1/1 Running 0 8s
When we now check the relevant parts of the routing table again we notice the addition of a bunch of rules:
$ sudo iptables-save | grep simpleservice
-A KUBE-SEP-ASIN52LB5SMYF6KR -s 172.17.0.2/32 -m comment --comment "namingthings/simpleservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-ASIN52LB5SMYF6KR -p tcp -m comment --comment "namingthings/simpleservice:" -m tcp -j DNAT --to-destination 172.17.0.2:9876
-A KUBE-SEP-RP53IYKEFRDLQANZ -s 172.17.0.4/32 -m comment --comment "namingthings/simpleservice:" -j KUBE-MARK-MASQ
-A KUBE-SEP-RP53IYKEFRDLQANZ -p tcp -m comment --comment "namingthings/simpleservice:" -m tcp -j DNAT --to-destination 172.17.0.4:9876
-A KUBE-SERVICES -d 172.30.40.155/32 -p tcp -m comment --comment "namingthings/simpleservice: cluster IP" -m tcp --dport 80 -j KUBE-SVC-IKIIGXZ2IBFIBYI6
-A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ASIN52LB5SMYF6KR
-A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -j KUBE-SEP-RP53IYKEFRDLQANZ
In above routing table listing we see rules for the newly created pod serving at 172.17.0.4:9876
as well as an additional rule -A KUBE-SVC-IKIIGXZ2IBFIBYI6 -m comment --comment "namingthings/simpleservice:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ASIN52LB5SMYF6KR
which causes the traffic to the service being equally split between our two pods by invoking the statistics
module of IPTables.
Summing up, this is our final setup:
Notice in above figure the different IP ranges for the pods, the service VIP as well as the node IP (192.168.99.100
in our example).
If you'd like to try out the above steps yourself, you can simply use the same setup I've used, that is, Minishift running locally on your machine. Ah, and before I forget it: kubectl
on my machine is actually a symlink to oc
, the OpenShift CLI client:
$ stat kubectl ; ls -l kubectl
File: "kubectl"
Size: 2 FileType: Symbolic Link
Mode: (0755/lrwxr-xr-x) Uid: ( 501/mhausenblas) Gid: ( 20/ staff)
Device: 1,4 Inode: 3685078 Links: 1
Access: Fri Apr 14 10:21:59 2017
Modify: Fri Apr 14 10:21:59 2017
Change: Fri Apr 14 10:21:59 2017
lrwxr-xr-x 1 mhausenblas staff 2 14 Apr 10:21 kubectl -> oc
Should you be interested in digging deeper into the topic of Kubernetes services and how to overcome scalability issues with large-scale service deployments—due to the inherent limitations of IPTables—check out the following resources:
- Kubernetes Services and Ingress Under X-ray
- Scale Kubernetes to Support 50,000 Services
- A series of networking-related posts by Wolfram Richter
- Networking concepts in OpenShift
I hope this post sheds some light on the topic of services in Kubernetes, what they do and how they work. Let me know what you think and which other Kubernetes objects we should have a look at in future posts.
저자 소개
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.