How to manipulate files with shell redirection and pipelines in Linux

Handling files, directories, command-line environments, and documentation is essential knowledge for sysadmins. Understanding file descriptors and how they relate to these topics improves your sysadmin skills. This article covers three standard Linux file descriptors for manipulating files through shell redirection and pipelines.

What are file descriptors?

In simple words, file descriptors are integers (numbers) that act as unique identifiers for an open file (or other I/O resource) in a Linux system. Remember, in Unix-like systems, "everything is a file descriptor or a process" (quoting Linus Torvalds), or even "everything can have a file descriptor" (quoting Neil Brown). It's important and useful to understand how the so-called three standard file descriptors, or standard streams, work because all processes use these channels for input and output operations.

To avoid making this article too long, I recommend you read in more detail the GNU documentation on Streams and File Descriptors, Descriptors and Streams, and Ports and File Descriptors. Here is a quick and simple overview of the concepts.

Consider the following image:

User interactions with the system are input through standard input (stdin), which is channel/stream 0, usually by using a keyboard. Then, any command executed through an interactive shell connects to a text terminal on which the shell is running and sends the output through either standard output (stdout), which is channel/stream 1, if it is OK, or through standard error (stderr), which is channel/stream 2 if it is not OK. The stdout is usually the terminal displayed by the monitor. There are other channels and streams (3 and up) that any process can use and don't have a default input or output.

[ Download the free guide to installing applications on Linux. ]

Shell I/O redirection

You can manipulate and change the default behavior of these three basic file descriptors by leveraging redirection and pipelines. For example, you can change your input from a keyboard to a file. Instead of getting messages in your terminal, you can redirect them to a file or even discard error messages instead of seeing them on your monitor. You can also redirect your output to the terminal and a file simultaneously. You may even process a command output as an input to another command.

There are three redirectors to work with: >, >>, and <. The following information describes each one:

Redirection with >

• command > file: Sends standard output to <file>
• command 2> file: Sends error output to <file>
• command 2>&1: Sends error output to standard output
• command > file 2>&1: Sends standard output and the error output to a file
• command &> file: Sends standard output and the error output to a file
• command 2>&1 > file: Sends error output to standard input and the standard input to a file

Append with >>

• command >> file: Appends standard output to a file
• command 2>> file: Appends error output to a file
• command >> file 2>&1: Appends standard output and error output to a file
• command &>> file: Appends standard output and error output to a file
• command 2>&1 >> file: Sends error output to standard input and appends standard input to a file

Redirect with <

• command < input: Feeds a command input from <input>
• command << input: Feeds a command or interactive program with a list defined by a delimiter; this is known as a here-document (heredoc)
• command <<< input: Feeds a command with <input>; this is known as a here-string

[ Download the free Bash shell scripting cheat sheet. ]

Shell I/O examples

Here are some examples of using each operator in the order presented above.

Redirect the standard output for a given command to a file:

$ echo "Enable Sysadmin" > myfile
$ cat myfile
Enable Sysadmin

Redirect error output for a given command to a file:

$ ls /root 2> myfile
$  cat myfile
ls: cannot open directory '/root': permission denied

Redirect error output for a given command to the standard output, the terminal:

$ ls /root 2>&1
ls: cannot open directory 'root/': Permission denied

Redirect both standard output and error output for a given command to a file:

$ find /usr -name ls > myfile 2>&1
$ cat myfile
/usr/bin/ls
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied
find: '/usr/libexec/initscripts/legacy-actions/auditd': Permission denied
$
$ find /usr -name ls &> myfile
$ cat myfile
/usr/bin/ls
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied
find: '/usr/libexec/initscripts/legacy-actions/auditd': Permission denied

Redirect error output for a given command to the standard output, the terminal, and the standard output for the same command to a file:

$ find /usr -name ls 2>&1 > myfile
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied
find: '/usr/libexec/initscripts/legacy-actions/auditd': Permission denied
$ cat myfile
/usr/bin/ls

Append the standard output for a given command to an existing file (if the file doesn’t exist, it creates it, just as the > operator does):

$ cat myfile
/usr/bin/ls
$ echo "Enable Sysadmin" >> myfile
$ cat myfile
/usr/bin/ls
Enable Sysadmin

Append the error output for a given command to an existing file:

$ ls /root 2>> myfile
$ cat myfile
/usr/bin/ls
Enable Sysadmin
ls: cannot open directory '/root': Permission denied

Append both the standard output and the error output for a given command to an existing file:

$ find /usr -name cd >> myfile 2>&1
$ cat myfile
/usr/bin/ls
Enable Sysadmin
ls: cannot open directory '/root': Permission denied
/usr/bin/cd
/usr/lib/.build-id/cd
/usr/share/X11/xkb/symbols/cd
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied
find: '/usr/libexec/initscripts/legacy-actions/auditd': Permission denied
$
$ find /usr -name cd &>> myfile
$ cat myfile
/usr/bin/ls
Enable Sysadmin
ls: cannot open directory '/root': Permission denied
/usr/bin/cd
/usr/lib/.build-id/cd
/usr/share/X11/xkb/symbols/cd
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied

Redirect the error output for a given command to the standard output, the terminal, and append the standard output for the same command to an existing file:

$ find /usr -name cd 2>&1 >> myfile
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied
find: '/usr/libexec/initscripts/legacy-actions/auditd': Permission denied
$ cat myfile
/usr/bin/ls
Enable Sysadmin
ls: cannot open directory '/root': Permission denied
/usr/bin/cd
/usr/lib/.build-id/cd
/usr/share/X11/xkb/symbols/cd
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied
find: '/usr/libexec/initscripts/legacy-actions/auditd': Permission denied
/usr/bin/cd
/usr/lib/.build-id/cd
/usr/share/X11/xkb/symbols/cd
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied

Feed a command with a nonstandard input, in this case, a file:

$ sort < myfile
/usr/bin/cd
/usr/bin/cd
/usr/bin/cd
/usr/bin/ls
/usr/bin/.build-id/cd
/usr/bin/.build-id/cd
/usr/bin/.build-id/cd
/usr/share/X11/xkb/symbols/cd
/usr/share/X11/xkb/symbols/cd
/usr/share/X11/xkb/symbols/cd
Enable Sysadmin
find: '/usr/libexec/initscripts/legacy-actions/auditd': Permission denied
find: '/usr/libexec/initscripts/legacy-actions/auditd': Permission denied
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/polkit-1/rules.d': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied
find: '/usr/share/selinux/targeted/default/active': Permission denied
ls: cannot open directory '/root': Permission denied

Feed a command with a non-standard dynamic input, in this case, the input is a list of strings needing a token delimiter, and also redirect the result to a file:

$ cat << EOF
> Enable
> Sysadmin
> EOF
Enable
Sysadmin
$
$ cat << EOF > myfile
> Enable
> Sysadmin
> EOF
$ cat myfile
Enable
Sysadmin
$

Feed a command with a nonstandard input, in this case, a string, with no need for a token delimiter:

$ read a b <<< "Sysadmin Enable"
$ echo $b $a
Enable Sysadmin

Shell piping

In Linux, a pipeline is a mechanism that allows two or more processes to be combined or executed concurrently. That means the process output will be handled as an input for the next one, and so on. It's not called a pipeline for nothing: It refers to the concept of a process flow being channeled through a pipe from a source to a destination.

[ Get the free Grep command cheat sheet. ]

The following image gives an example of how it works:

The operator used is the vertical bar on your keyboard, the | symbol. To "pipe" one command to another, separate the commands with this operator. It looks like this:

$ command1 | command2 | command3

Shell piping examples

Here are some examples:

Feed the grep input command with the output of the cat command:

$ cat /etc/passwd | grep localuser
localuser:×:1000:1000:Local User:/home/localuser/bin/bash

Redirect the standard output of the printf command to a file and then feed the sort command with the same newly created content into the file:

$ printf "Sysadmin\nEnable\n" > myfile | sort myfile
Enable
Sysadmin

Check the available RAM with the free command, filter the result for swap information with the grep command, and then use awk to cut only the the "total" column from the output to be exhibited:

$ free -h | grep Swap | awk '{print $2}'
1.0Gi
$

There are many other possibilities. You just need to understand your needs and use the pipeline accordingly to get what you want from these combinations.

Wrap up

As a sysadmin, it's crucial for you to understand the concepts behind file descriptors and know how to use the available operators to execute redirections and pipelining in a shell such as Bash so that you can manipulate the standard inputs and outputs on your system. This skill gives you a complete set of resources and possibilities to streamline, facilitate, improve, and automate the administration of your system. I hope this article aids you in understanding this topic, helps your Linux certification path, and adds to your general sysadmin knowledge.