Subscribe to the feed

Networking is a fundamental sysadmin skill, but it is often overlooked. Many sysadmins find networking topics challenging, and it can be difficult to progress from core network skills to advanced troubleshooting capabilities without regular practice. This may be why networking is a popular topic on Enable Sysadmin.

Kubernetes and its associated extensions, such as service meshes, introduce additional network complexity that an administrator must be prepared to tackle. Basic network tools, such as ping and traceroute, can be helpful during the initial troubleshooting stage. However, I've consistently found that viewing the packets traversing the wire is the best way to troubleshoot and understand complex protocol and application-level issues.

This article introduces ksniff, an excellent open source tool that I've been using. Ksniff is a plugin for kubectl that allows you to capture packets in your Kubernetes pods. I'll show you how to capture, filter, and save packets for later analysis. Let's get started!

Disclaimer: The techniques explained in this article should only be used in development environments and with a complete understanding of what you are doing. The official ksniff documentation does not yet recommend its use in production, and you should be aware that it uploads a precompiled tcpdump binary to your running pods.

Install ksniff and related tools

Ksniff is a plugin for kubectl, and you must install it before you can start using it. The official installation instructions for ksniff recommend using the Krew plugin manager, which is my preferred installation method. You can find installation instructions for Krew on the project's website. Once you have Krew installed, you can install ksniff with a single command:

$ kubectl krew install sniff
Updated the local copy of plugin index.
Installing plugin: sniff
Installed plugin: sniff
\
 | Use this plugin:
 | 	kubectl sniff
 | Documentation:
 | 	https://github.com/eldadru/ksniff
 | Caveats:
 | \
 |  | This plugin needs the following programs:
 |  | * wireshark (optional, used for live capture)
 | /
/
WARNING: You installed plugin "sniff" from the krew-index plugin repository.
   These plugins are not audited for security by the Krew maintainers.
   Run them at your own risk.

If you want to follow along and view captured packets in a graphical user interface (GUI) or terminal, you should also install Wireshark and tshark.

[ You might be wondering: Red Hat OpenShift and Kubernetes ... what's the difference? ]

Set up a workload to test

The best way to get started with ksniff is to perform a packet capture on a familiar workload, such as a basic web server. I recommend a simple Nginx pod, which you can run with a single command. These examples deploy a simple workload on a single-node K0s installation running on my workstation:

$ kubectl run --image=nginx nginx
pod/nginx created

$ kubectl get pod
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          4s

Once the pod is running, you can expose it as a NodePort service. While most admins generally avoid NodePort services in production, they provide a convenient way to test a service quickly:

$ kubectl expose pod nginx --port 80 --type=NodePort
service/nginx exposed

$ kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        2d5h
nginx        NodePort    10.99.139.251   <none>        80:32209/TCP   2s

Finally, you can confirm that the service is operable with a simple curl to the NodePort service:

$ kubectl get nodes -o wide
NAME            STATUS   ROLES           AGE    VERSION       INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
acritelli-k8s   Ready    control-plane   3d5h   v1.23.3+k0s   10.10.0.207   <none>        Ubuntu 20.04.4 LTS   5.14.0-1029-oem   containerd://1.5.9

$ curl 10.10.0.207:32209
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

[ Free online course: Red Hat Enterprise Linux technical overview. ]

Capture packets

Once you have installed ksniff and have a pod running, it's time to capture some network traffic. Running kubectl sniff $POD_NAME will begin a packet capture, launch Wireshark, and send the packet capture directly to Wireshark:

$ kubectl sniff nginx
INFO[0000] using tcpdump path at: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' 
INFO[0000] no container specified, taking first container we found in pod. 
INFO[0000] selected container: 'nginx'                  
INFO[0000] sniffing method: upload static tcpdump       
INFO[0000] sniffing on pod: 'nginx' [namespace: 'default', container: 'nginx', filter: '', interface: 'any'] 
INFO[0000] uploading static tcpdump binary from: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump' 
INFO[0000] uploading file: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'nginx' 
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'nginx', pod: 'nginx', namespace: 'default' 
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :'' 
INFO[0000] file found: ''                               
INFO[0000] file was already found on remote pod         
INFO[0000] tcpdump uploaded successfully                
INFO[0000] spawning wireshark!                          
INFO[0000] start sniffing on remote container           
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'nginx', pod: 'nginx', namespace: 'default'
ksniff results displayed in Wireshark

[ Get this free eBook: Managing your Kubernetes clusters for dummies. ]

If you prefer to stay entirely at the command-line interface (for example, if you are on a remote server), then you can also send ksniff's output to the tshark terminal program:

$ kubectl sniff nginx -o - | tshark -r -
INFO[0000] using tcpdump path at: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' 
INFO[0000] no container specified, taking first container we found in pod. 
INFO[0000] selected container: 'nginx'                  
INFO[0000] sniffing method: upload static tcpdump       
INFO[0000] sniffing on pod: 'nginx' [namespace: 'default', container: 'nginx', filter: '', interface: 'any'] 
INFO[0000] uploading static tcpdump binary from: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump' 
INFO[0000] uploading file: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'nginx' 
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'nginx', pod: 'nginx', namespace: 'default' 
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :'' 
INFO[0000] file found: ''                               
INFO[0000] file was already found on remote pod         
INFO[0000] tcpdump uploaded successfully                
INFO[0000] output file option specified, storing output in: '-' 
INFO[0000] start sniffing on remote container           
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'nginx', pod: 'nginx', namespace: 'default' 
    1   0.000000   10.244.0.1 → 10.244.0.18  TCP 76 29681 → 80 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=420487628 TSecr=0 WS=128
    2   0.000023  10.244.0.18 → 10.244.0.1   TCP 76 80 → 29681 [SYN, ACK] Seq=0 Ack=1 Win=64260 Len=0 MSS=1440 SACK_PERM=1 TSval=3356168433 TSecr=420487628 WS=128
    3   0.000040   10.244.0.1 → 10.244.0.18  TCP 68 29681 → 80 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=420487628 TSecr=3356168433
    4   0.000069   10.244.0.1 → 10.244.0.18  HTTP 149 GET / HTTP/1.1 
    5   0.000072  10.244.0.18 → 10.244.0.1   TCP 68 80 → 29681 [ACK] Seq=1 Ack=82 Win=64256 Len=0 TSval=3356168433 TSecr=420487628
    6   0.000160  10.244.0.18 → 10.244.0.1   TCP 306 HTTP/1.1 200 OK  [TCP segment of a reassembled PDU]
    7   0.000178   10.244.0.1 → 10.244.0.18  TCP 68 29681 → 80 [ACK] Seq=82 Ack=239 Win=65408 Len=0 TSval=420487628 TSecr=3356168433
    8   0.000192  10.244.0.18 → 10.244.0.1   HTTP 683 HTTP/1.1 200 OK  (text/html)
    9   0.000198   10.244.0.1 → 10.244.0.18  TCP 68 29681 → 80 [ACK] Seq=82 Ack=854 Win=64896 Len=0 TSval=420487628 TSecr=3356168433
   10   0.000296   10.244.0.1 → 10.244.0.18  TCP 68 29681 → 80 [FIN, ACK] Seq=82 Ack=854 Win=65536 Len=0 TSval=420487628 TSecr=3356168433
   11   0.000326  10.244.0.18 → 10.244.0.1   TCP 68 80 → 29681 [FIN, ACK] Seq=854 Ack=83 Win=64256 Len=0 TSval=3356168433 TSecr=420487628
   12   0.000354   10.244.0.1 → 10.244.0.18  TCP 68 29681 → 80 [ACK] Seq=83 Ack=855 Win=65536 Len=0 TSval=420487628 TSecr=3356168433
^C⏎  

Real-time analysis is useful, but saving a packet capture for later analysis is also common. This is especially helpful if you capture packets on a machine such as a jump host that does not have access to Wireshark or tshark. You can write packets to a file by specifying an output to ksniff. You can open the saved file in a protocol analyzer, such as Wireshark:

$ k sniff nginx -o /tmp/nginx_capture.pcap
INFO[0000] using tcpdump path at: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' 
INFO[0000] no container specified, taking first container we found in pod. 
INFO[0000] selected container: 'nginx'                  
INFO[0000] sniffing method: upload static tcpdump       
INFO[0000] sniffing on pod: 'nginx' [namespace: 'default', container: 'nginx', filter: '', interface: 'any'] 
INFO[0000] uploading static tcpdump binary from: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump' 
INFO[0000] uploading file: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'nginx' 
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'nginx', pod: 'nginx', namespace: 'default' 
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :'' 
INFO[0000] file found: ''                               
INFO[0000] file was already found on remote pod         
INFO[0000] tcpdump uploaded successfully                
INFO[0000] output file option specified, storing output in: '/tmp/nginx_capture.pcap' 
INFO[0000] start sniffing on remote container           
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'nginx', pod: 'nginx', namespace: 'default' 
^C⏎

$ file /tmp/nginx_capture.pcap 
/tmp/nginx_capture.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Linux cooked v1, capture length 262144)

Packet captures can become very messy, especially for pods that run complex or highly utilized workloads. The ability to apply tcpdump filters is critical to simplifying complex packet captures, and ksniff supports this out of the box:

$ kubectl sniff nginx -f "tcp port 80"

Wrap up

This article shows you how to leverage ksniff to capture packets inside of running Kubernetes pods. Packet captures provide a powerful method to observe and troubleshoot complex network and application issues, and their applicability is even more relevant as environments become increasingly complicated.

Performing packet captures is also a great way to sharpen your networking skills and develop a deeper understanding of complex environments, so I recommend you leverage a tool like ksniff at every available opportunity. Good luck, and happy networking!


About the author

Anthony Critelli is a Linux systems engineer with interests in automation, containerization, tracing, and performance. He started his professional career as a network engineer and eventually made the switch to the Linux systems side of IT. He holds a B.S. and an M.S. from the Rochester Institute of Technology.

Read full bio
UI_Icon-Red_Hat-Close-A-Black-RGB

Browse by channel

automation icon

Automation

The latest on IT automation for tech, teams, and environments

AI icon

Artificial intelligence

Updates on the platforms that free customers to run AI workloads anywhere

open hybrid cloud icon

Open hybrid cloud

Explore how we build a more flexible future with hybrid cloud

security icon

Security

The latest on how we reduce risks across environments and technologies

edge icon

Edge computing

Updates on the platforms that simplify operations at the edge

Infrastructure icon

Infrastructure

The latest on the world’s leading enterprise Linux platform

application development icon

Applications

Inside our solutions to the toughest application challenges

Original series icon

Original shows

Entertaining stories from the makers and leaders in enterprise tech