Working with pipes on the Linux command line

I'm sorry to inform you, but the command line didn't die off with the dinosaurs, nor did it disappear with the dodo or the carrier pigeon. The Linux command line is alive and well, and still going strong. It is an efficient way of quickly gathering and processing information, creating new scripts, and configuring systems. 

One of the most powerful shell operators is the pipe (|). The pipe takes output from one command and uses it as input for another. And, you're not limited to a single piped command—you can stack them as many times as you like, or until you run out of output or file descriptors.

One of the main purposes of piping is filtering. You use piping to filter the contents of a large file—to find a particular string or word, for example. This purpose is why the most popular use for pipes involves the commands grep and sort. But, you're not limited to those cases. You can pipe the output to any command that accepts stream input. Let's look at a theoretical example as an illustration of how this process works:

$ cmd1 | cmd2

Both cmd1 and cmd2 are command line utilities that output their results to the screen (stdout). When you pipe one command's output to another, however, the information from cmd1 doesn't produce output to the screen. The pipe redirects that output as input to cmd2. 

Note: Don't confuse pipe (|) redirection with file redirection (>) and (<). (An example of file redirection: cmd1 > file or cmd1 < file.) File redirection either sends output to a file instead of the screen, or pulls input from a file.

Let's look at some real-world examples of how piping works.

Checking on NICs

Let's say that you need to know if one of your network interface cards (NICs) has an IP address beginning with 192:

$ ifconfig | grep 192

inet 192.168.1.96  netmask 255.255.255.0  broadcast 192.168.1.255

You can also find out which live NICs you have on a system with a simple pipe to grep:

$ ifconfig | grep UP

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

You could also grep for "RUNNING" or "RUN" to display the same information.

Examining permissions

Maybe you want to find out how many directories under /etc are writeable by root:

$ sudo ls -lR | grep drwx

The results are too long to list here, but as you can see from your displayed list, there are a lot of them. You still need to find out how many there are, and a visual count would take a long time. An easy option is to pipe the results of your ls command to the wc (word count) command: 

$ sudo ls -lR | grep drwx | wc -l

285

The -l switch displays the number of lines. Your count might be different. This listing is from a fresh "RHEL 8 server no GUI" install.

[Want to try out Red Hat Enterprise Linux? Download it now for free.]

Counting files

You don't have to use grep all the time. For example, you can list the number of files in the /etc directory with this:

$ ls | wc -l

229

Again, your results might look different, but you know something is wrong if the command returns a small number of files.

Identifying processes

You can also perform complex tasks using pipes. To list the process IDs (PIDs) for all systemd-related processes:

$ ps -ef | grep systemd | awk '{ print $2 }'

1
471
500
608
631
7449
7494
17242

The awk command's $2 output isolates the second (PID) column from the ps command. Note that the last entry, PID 17242, is actually the PID for the grep command displaying this information, as you can see from the full listing results here:

khess    17242  7505  0 09:40 pts/0    00:00:00 grep --color=auto systemd

To remove this entry from your results, use the pipe operator again:

$ ps -ef | grep systemd | awk '{ print $2 }' | grep -v grep

The -v switch tells the grep command to invert or ignore lines that contain the string that follows—in this case any line containing the word "grep."

Sorting results

Another popular use of the pipe operator is to sort your results by piping to the sort command. Say that you want to sort the list of names in contacts.txt. First, let's look at the contents as they are in the file before sorting:

$ cat contacts.txt

Bob Jones
Leslie Smith
Dana David
Susan Gee
Leonard Schmidt
Linda Gray
Terry Jones
Colin Doe
Jenny Case

Now, sort the list:

$ cat contacts.txt | sort

Bob Jones
Colin Doe
Dana David
Jenny Case
Leonard Schmidt
Leslie Smith
Linda Gray
Susan Gee
Terry Jones

You can reverse the sort order with the -r switch:

$ cat contacts.txt | sort -r

Terry Jones
Susan Gee
Linda Gray
Leslie Smith
Leonard Schmidt
Jenny Case
Dana David
Colin Doe
Bob Jones

Was the output for either of these what you expected? By default, the sort command performs a dictionary sort on the first word or column, which is why Bob Jones and Terry Jones are not listed one after the other.

You can sort by the second column with the -k switch, specifying the column you want to sort on, which in our list is column two:

$ cat contacts.txt | sort -k2

Jenny Case
Dana David
Colin Doe
Susan Gee
Linda Gray
Bob Jones
Terry Jones
Leonard Schmidt
Leslie Smith

If you have a long contact list and you think that this list contains duplicate entries, you can pipe your list to sort and then uniq to remove those duplicates:

$ cat contacts.txt | sort | uniq

This command only displays unique entries, but it doesn't save the results. To sort the file, filter unique entries, and then save the new file, use this:

$ cat contacts.txt | sort -k2 | uniq > contact_list.txt

Remember that the pipe operator and the file redirect operators do different things.

Wrapping up

Now that you've had a taste of the command line's power, do you think you can handle the responsibility? Command line utilities have more flexibility than their graphical counterparts do. And for some tasks, there are no graphical equivalents.

The best way to learn command line behavior is to experiment. For example, the sort command didn't do what you thought it should until you explored further. Don't forget to use the power of your manual (man) pages to find those "hidden" secret switches and options that turn command line frustration into command line success.