Purging the process, Part 2
Advanced topics in pipes, filters, and redirection
Last month I covered several basics, such as input redirection:
$ grep "hello" <hello.txt say hello.
| Purging the process: Read the whole series! | |
|---|---|
|
$ grep "hello" >junk.txt Now is the time for every good person to say hello. (type control-D here) $ cat junk.txt say hello. $
Input and output redirection, and the use of input files on the command line instead of redirected input:
$ grep "Now" <hello.txt >junk.txt $ grep "Now" hello.txt >junk.txt
Appending additional data to a file using an output redirection:
$ echo "Now is the time" >hello.txt $ echo "for every good person to" >>hello.txt $ echo "say hello." >>hello.txt $ cat hello.txt Now is the time for every good person to say hello. $
Redirecting standard output and standard error, and redirecting standard error to the
/dev/null byte wastebasket: $ find / -name *.txt -exec ls -l {} \; 2>/dev/null >textfiles
$ Basic pipes:
$ grep "hello" < hello.txt | sed -e "s/hello/bye/" > result.txt $( grep "hello" | sed -e "s/hello/bye/" ) < hello.txt > result.txt $
I also stated that redirecting output to an existing file would delete the file and create a new version of it. In the following example, the fourth line causes
hello.txt to be overwritten with a new version of the file containing only a single line, bye. $ echo "hello" >hello.txt $ cat hello.txt hello $ echo "bye" >hello.txt $ cat hello.txt bye
You can set the
noclobber option to prevent redirected files from automatically overwriting their predecessors. In the following example, the option causes an error message at line six when the user tries to overwrite the hello.txt file. $ set noclobber $ echo "hello" >hello.txt $ cat hello.txt hello $ echo "bye" >hello.txt File "hello.txt" already exists $ cat hello.txt hello unset noclobber
If
noclobber is set, you can force a redirection to clobber any pre-existing file by using the >| redirection operator. This operator looks like a redirection to a pipe, but it's actually just a force redirect to override the noclobber option. In the following example the forced redirection operator prevents any error messages. $ set noclobber $ echo "hello" >|hello.txt $ cat hello.txt hello $ echo "bye" >|hello.txt $ cat hello.txt bye unset noclobber
Combining standard output and standard error
Redirection is frequently used for jobs that run for a long period of time, or for jobs that produce a lot of output. For such jobs, redirection can capture the results in a file. When this is done, it's also necessary to capture any output errors. Remember that if you redirect standard output but not standard error, output will go to a file and error messages will still go to your screen. The following
Redirection is frequently used for jobs that run for a long period of time, or for jobs that produce a lot of output. For such jobs, redirection can capture the results in a file. When this is done, it's also necessary to capture any output errors. Remember that if you redirect standard output but not standard error, output will go to a file and error messages will still go to your screen. The following
find command will save the results to found.txt, although errors still appear on the screen. $ find / -name *.txt -exec ls -l {} \; >found.txt
find: /some/directory: Permission denied
find: /another/one: Permission denied
$ The redirection operator is actually a number followed by the redirection symbol, as in the following example. If number is omitted, 1 is the default.
$ find / -name *.txt -exec ls -l {} \; 1>found.txt
$ The following commands are equivalent:
$ find / -name *.txt -exec ls -l {} \; 1>found.txt
$ find / -name *.txt -exec ls -l {} \; >found.txt
$ Unix utilities open three files automatically when a program starts up. These files are given file descriptor numbers inside the program -- 0, 1, and 2 -- but they're more commonly known as stdin (standard input -- file descriptor 0), stdout (standard output -- file descriptor 1), and stderr (standard error -- file descriptor 2). When the program starts, default assignments for these files are made to
/dev/tty, which is the device name for your terminal. The stdin file is assigned to the keyboard of your terminal, while stdout and stderr are assigned to the screen of your terminal. The output redirection operator defaults to 1; thus > and 1> are equivalent. The input redirection operators < and <0 are equivalent. Redirecting standard error, file descriptor 2, requires that its number be explicitly included in the redirection symbol. The following examples use
1> to redirect standard output because it helps clarify how the redirection works. When reviewing these examples remember that > and 1> are the same. One method of handling the logging problem would be to create separate logs for each of the outputs, as in the following example.
$ find / -name *.txt -exec ls -l {} \; 1>found.txt 2>errors.txt
$ It is also possible to redirect an output by attaching it to an already open redirection using the
>& redirection operator. In the following example, the standard output of find is redirected to the file result.txt. The 2>&1 redirection command instructs the shell to attach the output from standard error (2) to the output of standard output (1). Now both standard output and standard error are sent to result.txt. $ find / -name *.txt -exec ls -l {} \; 1>result.txt 2>&1
$ The order of redirection is important. In the following example, the output of file descriptor 2 (standard error) is attached to file descriptor 1. At this point, standard output is still attached to the terminal, so standard error is sent to the terminal. The next redirection sends standard output to
result.txt. This redirection doesn't drag file descriptor 2 along with it, so standard error is left pointing to the terminal device. $ find / -name *.txt -exec ls -l {} \; 2>&1 1>result.txt
find: /some/directory: Permission denied
find: /another/one: Permission denied
$ Input redirection from here documents
Perhaps one of the most useful forms of redirection is redirecting input from a
Perhaps one of the most useful forms of redirection is redirecting input from a
here document. A shell script can be written that executes a command and serves all input to the command. This is frequently used for a command that is normally run interactively. As an extreme example, I will show you how to do this with the editor vi. I am using vi for two reasons: first, it's interactive, and second, you're probably fairly familiar with it already and so will have a better understanding of what the script's doing. Normally, hands-off editing is done with the sed command. First, create a text file with several
hello strings in it, as in the following example, then name it hello.txt. sample hello.txt hello world hello broadway hello dolly
Create a file named
here.sh that contains the lines in the example below. The second line starts the vi editor on the hello.txt file and the <<END-OF-INPUT option states that vi will run taking its input from this current file, here.sh, reading in a line at a time until a single line containing END-OF-INPUT is read in. The subsequent lines are vi commands to globally search for hello, replace each instance of it with bye, write the file back out, then quit. The next line is the END-OF-INPUT line and final echo statement to indicate that the editing is complete. # here.sh - sample here document vi hello.txt <<END-OF-INPUT :g/hello/s//bye/g :w :q! END-OF-INPUT echo "Editing complete"
Change the mode on the file to make it executable:
$ chmod a+x here.sh
When you execute the
here.sh script, you may receive a warning from vi that it's not running in interactive mode. Next, the actual editing takes place; afterwards, you can cat out the hello.txt file and see your handiwork. $ ./here.sh Vim: Warning: Input is not from a terminal Editing complete $ cat hello.txt sample bye.txt bye world bye broadway bye dolly
If you really want to suppress the vi warning, redirect the error to the
/dev/null device, as in the following version of here.sh: # here.sh - sample here document vi hello.txt 2>/dev/null <<END-OF-INPUT :g/hello/s//bye/g :w :q! END-OF-INPUT echo "Editing complete"
here documents frequently appear as small pieces of larger scripts. In order to make the here portion stand out, it's helpful to indent that section of the shell. Using a minus (-) in front of the end-of-input marker eats the white spaces at the beginning of a line and prevents them from being passed on to the program. The following is an example: # here.sh - sample here document vi hello.txt 2>/dev/null <<-STOP-HERE :g/hello/s//bye/g :w :q! STOP-HERE echo "Editing complete"
Because it's an interactive program, the
ftp utility is a common candidate for here document status. The following example starts ftp and redirects standard output and standard error to xfr.log. The process logs in to a remote system named nj_system, switches to binary transfer mode, creates two directories, transfers a file named newstuff.a to the remote system, and signs out again. Using a here document makes it possible to execute ftp through a shell script while seeing what the script is doing. The second example below is another method of doing this, but it involves a separate file with the ftp commands. # xfr.sh - Transfers to a remote system
district=nj
ftplog=xfr.log
insbase=/usr/installations
insdir=$insbase/new
inskit=newstuff.a
echo "Transferring to" $district
ftp 1>>$ftplog 2>&1 $district"_system" <<-ALL-DONE
user mo ddd789
binary
mkdir $insbase
chmod 777 $insbase
mkdir "$insdir"
chmod 777 $insdir
put $inskit $insdir/$inskit
chmod 777 $insdir/$inskit
bye
ALL-DONE
echo "Transfer to" $district "complete." The first file would have to contain nothing but the commands for
ftp, and couldn't take advantage of script variables. Here's a sample input for ftp: user mo ddd789 binary mkdir /usr/installations chmod 777 /usr/installations mkdir /usr/installations/new chmod 777 /usr/installations/new put newstuff.a /usr/installations/new/newstuff.a chmod 777 /usr/installations/new /newstuff.a bye # xfr.sh - Transfers to a remote system district=nj ftplog=xfr.log echo "Transferring to" $district ftp 1>>$ftplog 2>&1 $district"_system" <ftp_commands echo "Transfer to" $district "complete."
In our next installment, I'll cover Unix system and global variables. What are they and how do you use them? I have been meaning to do this one for a while, and now seems like a good time.
