3.1. Redirection and Pipes

One of the interesting ideas that pervades Unix is that many, if not most, programs can be viewed as filters or transforms that take a stream of text as input and produce an altered stream of text as output. Many Unix commands are designed to perform relatively trivial tasks, perhaps not very useful by themselves, that can be chained together in interesting and useful ways.

The practical consequence of this is that Unix shells devote special attention to a standard input stream that forms the main input to most programs/commands, and to a standard output stream that forms the main output from most programs/commands.[19] The shell attempts to make it easy either to redirect one of these standard streams to a file or to pipe the standard output stream of one program into the standard input of another.

Redirection

For example, the program wc (for word count) reads text from its input stream and produces as its output stream three numbers indicating the number of lines, words, and characters that it saw. You could invoke this directly:

   wc
   Hello.
   How are you?
   ^D

in which case, you would see as output:

     2     4    20

For this to be very useful, however, we need to make it accept a file as input. This is done by using the < operator in the shell. Think of the < as an arrow indicating data flowing towards the command from a filename: If hello.c is this file:

#include <stdio.h> 
int main () 
{
  printf ("Hello from C!\n"); 
  return 0;
}

then the command

wc < hello.c

produces the output

     6    13    80

On the output end, the shell operator > redirects the standard output into a file (again, think of this as an arrow indicating data flowing into a filename from the command):

wc < hello.c > hello.wc

produces no output on the screen, but creates a file called hello.wc. That file will contain the output

     6    13    80

of the wc command.

The output redirection operator has a couple of important variants. First, the shell generally does not allow you to redirect into an existing file. If you give the command

   wc < hello.c > hello.wc

a second time, the shell will refuse to perform the command. You can force the shell to delete an existing file and create a new one for redirection by changing the > to >!.

Example 2.29. Try This:

cd ~/playing
echo Hello > greeting.txt
more greeting.txt
echo Goodbye > greeting.txt
more greeting.txt
echo Farewell >! greeting.txt
more greeting.txt

Second, sometimes we would like to add output to the end of an existing file instead of replacing that file. This is done with the operator >>. So the code sequence

   wc < hello.c >! hello.wc
   wc < hello.c >> hello.wc

would result in a file hello.wc with contents

     6    13    80
     6    13    80

regardless of whether hello.wc had existed previously.

Example 2.30. Try This:

cd ~/playing
echo Aloha >> greeting.txt
more greeting.txt

Pipes

To pipe the output of one command into the input of another, use the shell operator |. A common example of a pipe is to take a command that may have a large amount of output and to pipe it through more to facilitate viewing.

Example 2.31. Try This:

   ls /bin | more


As you gain facility with a greater variety of Unix text manipulation commands, you will find that redirection and pipes can be a powerful combination. For example, suppose that you have written program myprog that emits a great deal of output, among which might be some error messages starting with the phrase ERROR: . If you wanted to read only the error messages, you could, of course, just view all the output, watching for the occasional error message:

   myprog | more

But if the program produces a lot of output, this will quickly become tedious. However, we previously encountered the program grep, which scans its input stream, printing only those lines matching a given regular expression. By piping the myprog output through grep, we can limit the output to the part we really want to see:

   myprog | grep "ERROR:" | more



[19] There is actually a second output stream supported by many programs, the standard error stream, used for writing error/debugging messages.