How to use the lsof command to troubleshoot Linux

One of the things I appreciate most about Linux is the chance to explore my operating system. Nothing is off limits in Linux. That's a huge opportunity—a very comfortable benefit—for anyone using Linux daily, whether you're a sysadmin, developer, network admin, or hobbyist eager to learn more.

It has real-world ramifications, too, because sometimes you really do need to probe into the inner workings of a system to understand a problem. Every sysadmin knows the feeling of getting a vague message like "The server is slow" or "This application isn't responding." That lucky sysadmin gets to perform a "treasure hunt" to find the problem. However, Linux provides all the tools you need.

[Cheat sheet: Old Linux commands and their modern replacements ]

The lsof command, in combination with other tools like top or ps, can be very helpful when diagnosing a potential bottleneck or similar issue.

List open files

The lsof command is an acronym for "list open files," but its potential isn't limited to just that role. It's commonly said that in Linux, everything is a file. In many ways, that's true, so a utility that lists open files is actually pretty useful. The lsof utility is a robust interface for the information inside the /proc virtual filesystem.

For example, suppose you need to know what process is using a particular directory:

$ lsof /run
COMMAND    PID  USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
systemd      1  root   31u  FIFO   0,24      0t0 19314 /run/initctl
systemd      1  root   38u  FIFO   0,24      0t0 19331 /run/dmeventd-server
systemd      1  root   39u  FIFO   0,24      0t0 19332 /run/dmeventd-client
systemd-j  713  root  mem    REG   0,24  8388608 12871 /run/log/journal/aaff550ef54147ddb4d1776d86b04055/system.journal
NetworkMa  881  root   20w  FIFO   0,24      0t0 24896 /run/systemd/inhibit/1.ref
rhsmcertd  906  root    3wW  REG   0,24        0 22479 /run/lock/subsys/rhsmcertd
crond      910  root    3uW  REG   0,24        4 24002 /run/crond.pid
rsyslogd  1224  root  mem    REG   0,24  8388608 12871 /run/log/journal/aaff550ef54147ddb4d1776d86b04055/system.journal
rsyslogd  1224  root   10r   REG   0,24  8388608 12871 /run/log/journal/aaff550ef54147ddb4d1776d86b04055/system.journal
sshd      1651  root   11w  FIFO   0,24      0t0 27390 /run/systemd/sessions/1.ref
sshd      1666 admin   11w  FIFO   0,24      0t0 27390 /run/systemd/sessions/1.ref

Or maybe you need the exact opposite. Perhaps you need to know what files a particular process has open. This example looks at sshd:

$ lsof -p 890
CMD   PID USER   FD  TYPE DEVICE SIZE/OFF     NODE NAME
sshd  890 root  cwd  DIR  253,0      224      128 /
sshd  890 root  rtd  DIR  253,0      224      128 /
sshd  890 root  txt  REG  253,0   886344   646480 /usr/sbin/sshd
sshd  890 root  mem  REG  253,0  6940392 34389337 /var/lib/sss/mc/group
sshd  890 root  mem  REG  253,0  9253600 34389336 /var/lib/sss/mc/passwd
sshd  890 root  mem  REG  253,0    46312   570898 ...libnss_sss.so
sshd  890 root  mem  REG  253,0   144392     1847 ...libgpg-error.so
[...]
sshd  890 root    0r CHR    1,3      0t0     1027 /dev/null
sshd  890 root    1u unix 0xff...    0t0    22395 type=STREAM
sshd  890 root    2u unix 0xff...    0t0    22395 type=STREAM
sshd  890 root    4u unix 0xff...    0t0    24654 type=STREAM
sshd  890 root    5u IPv4 24658      0t0      TCP *:ssh (LISTEN)
sshd  890 root    6r REG  253,0  6940392 34389337 /var/lib/sss/mc/group
sshd  890 root    7u IPv6 24666      0t0      TCP *:ssh (LISTEN)

Skip to the bottom of list

Image

Download now

Here's an explanation of each column:

• COMMAND: The command name
• PID: Process ID (PID) of the process
• USER: Owner of the process
• FD: File descriptor definition
• TYPE: Type of file descriptor
• DEVICE: Device number or, in the case of a block device, character or other
• SIZE/OFF: Dimension of the file or offset (the suffix 0t is the offset)
• NODE: Node description of the local file; this could be the number of the local file, TCP, UDP, or STR (stream)
• NAME: The name of the mount point where the file resides

Open files

You can also discover what files a particular user has open:

$ lsof -u admin

This information is especially useful when you can't unmount an external device because a file on it is being actively accessed.

[ Get the guide to installing applications on Linux. ]

Network debugging

As I've said, everything on Linux is a file, so lsof isn't limited to the local filesystem. You can also use it for network debugging.

For example, suppose you need to know what process uses a particular TCP port (like 22, for example):

$ lsof -i TCP:22
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd     890  root    5u  IPv4  24658      0t0  TCP *:ssh (LISTEN)
sshd     890  root    7u  IPv6  24666      0t0  TCP *:ssh (LISTEN)

You can also obtain a report on all network processes associated with a particular interface:

$ lsof -i TCP@127.0.0.1
COMMAND PID USER   FD TYPE  DEVICE SZ/OFF NODE NAME
cupsd   4408 root  8u IPv4  32331   0t0   TCP  ...ipp (LISTEN)
qemu-s 13462 qemu 11u IPv4  98536   0t0   TCP  ...rfb (LISTEN)

[ For more tips on Linux administration, read How to analyze a Linux process' memory map with pmap. ]

Great potential

Despite its seemingly humble purpose, the lsof tool is a powerful tool on a system that can treat almost anything as a file. Read the lsof man page to get more ideas of how this simple utility can help you perform some complex tasks.