Packet sniffer basics for network troubleshooting

Performing packet captures using a sniffer can be an extremely powerful method for diagnosing complex issues. When all else fails, it’s often helpful to view the raw data being sent across the wire. The packets don’t lie, and analyzing your application traffic at a low level can reveal deeper symptoms of a problem (or reveal a problem that you didn’t even know existed).

In this article, we’ll cover some fundamentals of using the CLI-based tcpdump tool.

Sniffer basics

So first things first: What do we mean when we say "packet sniffer?" A packet sniffer is simply a piece of software that allows you to capture packets on your network. Tcpdump and Wireshark are examples of packet sniffers. Tcpdump provides a CLI packet sniffer, and Wireshark provides a feature-rich GUI for sniffing and analyzing packets.

By default, tcpdump operates in promiscuous mode. This simply means that all packets reaching a host will be sent to tcpdump for inspection. This setting even includes traffic that was not destined for the specific host that you are capturing on, such as broadcast and multicast traffic. Of course, tcpdump isn’t some magical piece of software: It can only capture those packets that somehow reach one of the physical interfaces on your machine.

Installing tcpdump is simple. It’s available in the standard package repositories on your Red Hat system, and you can install it by name:

 # yum install -y tcpdump

Common packet sniffing scenarios

Capturing all of the traffic coming into your machine may sound conceptually cool, but it also sounds fairly low level for many of the activities that we perform in our day-to-day work as sysadmins. So, when would you use a packet capturing tool? I typically turn to a packet sniffer when I’m troubleshooting a network application issue and I’ve exhausted all other options. Often, I have already performed basic network troubleshooting and reviewed any application log files, but I still can’t get to the bottom of an issue. At this point, breaking out a packet sniffer to observe the actual data being sent on the wire can be instructive.

Another great use case for a packet sniffer is pedagogical. Watching the packets involved in an application exchange can go a long way toward improving your understanding of the underlying protocols. For example, it can be invaluable to observe the full packet flow of a recursive DNS query when trying to understand how DNS works.

Performing basic packet captures

The best way to learn is by just diving in, so let’s get started with some basic packet captures. First, let’s try out tcpdump without any special options. Note that you’ll need to be the superuser to perform packet captures (technically, you can run it from a regular account with special capabilities, but it’s usually easier to run it as root). Use Ctrl+C, or send a SIGTERM to the tcpdump process ID (PID) to stop the capture.

Default tcpdump

To see the default output of tcpdump, simply type the command:

# tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:41:59.878706 IP localhost.localdomain > lga34s19-in-f14.1e100.net: ICMP echo request, id 3298, seq 5, length 64
20:41:59.901298 IP lga34s19-in-f14.1e100.net > localhost.localdomain: ICMP echo reply, id 3298, seq 5, length 64
20:42:00.878245 IP localhost.localdomain > lga34s19-in-f14.1e100.net: ICMP echo request, id 3298, seq 6, length 64
20:42:00.888530 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:11:23:84.8002, length 35
20:42:00.901329 IP lga34s19-in-f14.1e100.net > localhost.localdomain: ICMP echo reply, id 3298, seq 6, length 64
20:42:01.112737 ARP, Request who-has localhost.localdomain tell acritelli-laptop, length 28
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

Note: Rather than having a lot of output fly by before pressing Ctrl+C, you can specify how many packets you want to see with the -c flag. The command above could instead be tcpdump -c 6 to get the same results (six packets captured).

The output from tcpdump can be a bit daunting at first, but you’ll get used to looking at it after you’ve used this tool a few times. Let’s break down the fields, from left to right:

• Timestamp
• Protocol, such as IP
• Source (localhost.localdomain)
• Destination (lga34s19-in-f14.1e100.net)
• Protocol-specific fields (e.g., ICMP echo reply)
• Basic protocol information (e.g., sequence number and length)

Sniffing a specific interface

Notice that the top of the previous example’s output shows you the interface that tcpdump starts capturing on (eth0), and the bottom of the capture includes summary statistics about the captured packets.

The first thing that you’ll probably want to do when using tcpdump is to specify a particular interface for performing captures. By default, tcpdump will pick the lowest numbered interface that is "up." Many servers have multiple interfaces, and you’ll want to be explicit about the interface that you use for capturing. Additionally, some "special" interface types, such as a netfilter interface, may float to the top of the list. This behavior can cause confusion, so it’s best to specify the interface that you’re interested in.

Let’s start by viewing the interfaces that are available for capturing:

# tcpdump --list-interfaces
1.eth0
2.nflog (Linux netfilter log (NFLOG) interface)
3.nfqueue (Linux netfilter queue (NFQUEUE) interface)
4.usbmon1 (USB bus number 1)
5.usbmon2 (USB bus number 2)
6.usbmon3 (USB bus number 3)
7.usbmon4 (USB bus number 4)
8.ens9
9.any (Pseudo-device that captures on all interfaces)
10.lo [Loopback]

With a list of interfaces at our disposal, we can now specify the interface to listen to with the -i flag. Note that either the interface name or the number from the --list-interfaces command can be used:

# tcpdump -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:50:00.858552 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:11:23:84.8002, length 35
20:50:02.841866 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:11:23:84.8002, length 35
20:50:04.858167 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:11:23:84.8002, length 35
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

Looking at the above captures provides us with basic information about the packets traversing our network. It looks like these packets contain Spanning Tree Protocol (STP) output, perhaps from an upstream switch. Technically, these aren’t packets, they’re layer two frames. However, you’ll hear the terms used interchangeably when discussing packet captures.

Getting more information

The previous example’s simple troubleshooting result can be great for identifying obvious issues, but sometimes we need more information to really dig into a complex problem. Knowing how to adjust the verbosity of your capture is important, as it allows you to dig deeper into the actual data contained within the packets.

The verbosity level of tcpdump is controlled by appending between one and three -v flags to the command:

# tcpdump -i eth0 -vvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:49:22.844765 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:11:23:84.8002, length 35
         message-age 0.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 2.00s
         root-id 8000.52:54:00:11:23:84, root-pathcost 0
20:49:24.860576 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:11:23:84.8002, length 35
         message-age 0.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 2.00s
         root-id 8000.52:54:00:11:23:84, root-pathcost 0
20:49:26.843910 STP 802.1d, Config, Flags [none], bridge-id 8000.52:54:00:11:23:84.8002, length 35
         message-age 0.00s, max-age 20.00s, hello-time 2.00s, forwarding-delay 2.00s
         root-id 8000.52:54:00:11:23:84, root-pathcost 0
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

Note that by specifying the maximum level of verbosity, I can see far more information about the packet body. Above, we can see additional information about the data in the STP packet, such as the root bridge ID and root path cost. If you’re not familiar with STP, don’t worry about it. The important thing to notice here is that by increasing the verbosity, we can get additional information about our network traffic.

Seeing a packet’s exact bytes

Increasing the verbosity is handy, but we’re still not seeing the meat of the packets’ contents. If you really want to see the exact bytes that are in the packets, you can use the -x and -X flags. The -x flag prints the data in each packet in hex, while the -X flag also prints the data in ASCII. Here is the output using -x:

# tcpdump -i ens9 -c 1 -x port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens9, link-type EN10MB (Ethernet), capture size 262144 bytes
21:27:41.121804 IP localhost.localdomain.55310 > dns.google.domain: 8656+ [1au] A? www.google.com. (43)
      0x0000: 4500 0047 ff25 0000 4011 302b c0a8 7a9d
      0x0010: 0808 0808 d80e 0035 0033 0246 21d0 0120
      0x0020: 0001 0000 0000 0001 0377 7777 0667 6f6f
      0x0030: 676c 6503 636f 6d00 0001 0001 0000 2910
      0x0040: 0000 0000 0000 00
1 packet captured
5 packets received by filter
0 packets dropped by kernel

And here is the output using -X:

# tcpdump -i ens9 -c 1 -X port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens9, link-type EN10MB (Ethernet), capture size 262144 bytes
21:27:47.269762 IP localhost.localdomain.38787 > dns.google.domain: 48705+ [1au] A? www.google.com. (43)
      0x0000: 4500 0047 0a9c 0000 4011 24b5 c0a8 7a9d E..G....@.$...z.
      0x0010: 0808 0808 9783 0035 0033 a65f be41 0120 .......5.3._.A..
      0x0020: 0001 0000 0000 0001 0377 7777 0667 6f6f .........www.goo
      0x0030: 676c 6503 636f 6d00 0001 0001 0000 2910 gle.com.......).
      0x0040: 0000 0000 0000 00 .......
1 packet captured
5 packets received by filter
0 packets dropped by kernel

Notice the hex and ASCII data in the above examples. In this case, I was performing a DNS A record query for www.google.com. Also, note that I used the -c flag to specify the number of packets to capture, and I provided a capture filter of port 53. We’ll discuss filters in the next article.

Sticking with the numbers

As you’re working with tcpdump, you may notice that its default behavior is to automatically resolve IP addresses to fully qualified domain names (FQDNs). Tcpdump will also translate port numbers (such as 22) into friendly names (such as SSH). While this behavior is nice, we often want to see the numeric data so that we don’t obscure our troubleshooting in any way. This default behavior can be changed by passing either -n to disable IP address lookups, or -nn to disable both IP address and port lookups.

Here is the result when you use -nn:

# tcpdump -i ens9 -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens9, link-type EN10MB (Ethernet), capture size 262144 bytes
20:57:50.004451 IP 192.168.122.157.22 > 192.168.122.1.49540: Flags [P.], seq 2961250669:2961250889, ack 823295439, win 313, options [nop,nop,TS val 1210110 ecr 672601542], length 220
20:57:50.004805 IP 192.168.122.1.49540 > 192.168.122.157.22: Flags [.], ack 220, win 1444, options [nop,nop,TS val 672601548 ecr 1210110], length 0
20:57:50.005229 IP 192.168.122.157.22 > 192.168.122.1.49540: Flags [P.], seq 220:624, ack 1, win 313, options [nop,nop,TS val 1210111 ecr 672601548], length 404
20:57:50.005489 IP 192.168.122.1.49540 > 192.168.122.157.22: Flags [.], ack 624, win 1444, options [nop,nop,TS val 672601549 ecr 1210111], length 0
20:57:50.005772 IP 192.168.122.157.22 > 192.168.122.1.49540: Flags [P.], seq 624:1004, ack 1, win 313, options [nop,nop,TS val 1210111 ecr 672601549], length 380
20:57:50.006040 IP 192.168.122.1.49540 > 192.168.122.157.22: Flags [.], ack 1004, win 1444, options [nop,nop,TS val 672601549 ecr 1210111], length 0
20:57:50.006341 IP 192.168.122.157.22 > 192.168.122.1.49540: Flags [P.], seq 1004:1384, ack 1, win 313, options [nop,nop,TS val 1210112 ecr 672601549], length 380

Notice that in the above output, we can see that the IP addresses (192.168.122.157 and 192.168.122.1) remain untouched. The same is true for the ports (22 and 49540).

Saving your dump

At some point, you may want to save your packet captures for later analysis, or for deeper analysis with a graphical tool like Wireshark. This task can be easily accomplished with the -w flag, which allows you to write a packet capture file:

# tcpdump -i ens9 -w my_packet_capture.pcap
tcpdump: listening on ens9, link-type EN10MB (Ethernet), capture size 262144 bytes
^C5 packets captured
7 packets received by filter
0 packets dropped by kernel

# file my_packet_capture.pcap
my_packet_capture.pcap: tcpdump capture file (little-endian) - version 2.4 (Ethernet, capture length 262144)

Note: You can read your saved file using tcpdump with the -r flag, or with a different application that supports the pcap file format.

As we can see above, the -w flag produced a handy pcap file that we can take with us.

Wrapping up

Performing packet captures is a powerful technique in your network troubleshooting skills inventory, especially when you’re stuck on an issue and the rest of the network looks OK. Understanding how to use tcpdump on the command line can save you hours of frustration when trying to solve network application issues, and the syntax is quite intuitive once you get used to it.

[Need more on networking? Download the Linux networking cheat sheet.]