Today


Wednesday, February 9, 2011

Understanding Unix shells and environment variables Part 1

A shell variable is a memory storage area that can be used to hold a value, which can then be used by any built-in shell command within a single shell. An environment variable is a shell variable that has been exported or published to the environment by a shell command so that shells and shell scripts executed below the parent shell also have access to the variable.
 Unix shells and environment variables: Read the whole series! 
One built-in shell command can set a shell variable value, while another can pick it up. In the following doecho script example, $PLACE is set in the first line and picked up in the second line by the built-in echo command.
Create this script and save it as doecho. Change the mode using chmod a+x doecho:
# doecho sample variable
PLACE=Hollywood
echo "doecho says Hello " $PLACE
Run the program as shown below.
In all of the following examples, I use the convention of ./command to execute a shell script in the current directory. You don't need to do this if your $PATH variable contains the . as one of the searched directories. The ./command method works for scripts in your current directory, even if the current directory isn't included on your path.
$ ./doecho
doecho says Hello Hollywood
$
In this first example, $PLACE is a shell variable.
Now, create another shell script called echoplace and change its mode to executable.
# echoplace echo $PLACE variable
echo "echoplace says Hello " $PLACE
Modify doecho to execute echoplace as its last step.
# doecho sample variable
PLACE=Hollywood
echo "doecho says Hello " $PLACE
./echoplace
Run the doecho script. The output is a bit surprising.
$ ./doecho
doecho says Hello Hollywood
echoplace says Hello
$
In this example, echoplace is run as the last command of doecho. It tries to echo the $PLACE variable but comes up blank. Say goodbye to Hollywood.
To understand what happened here you need understand something about shell invocation -- the sequence of events that occur when you run a shell or shell script. When a shell begins to execute any command, it checks to see if the command is built-in (like echo), an executable program (like vi or grep), a user-defined function, or an executable shell script. If it's any of the first three, it directly executes the command, function, or program; but if the command is an executable shell script, the shell spawns another running copy of itself -- a child shell. The spawned child shell uses the shell script as an input file and reads it in line by line as commands to execute.
When you type ./doecho to execute the doecho script, you're actually executing a command that is something like one of the following, depending on which shell you're using. (See the Resources section at the end of this column for more information on redirection.)
$ sh < ./doecho
            (or)
$ ksh <./doecho
The new shell, spawned as a child of your starting-level shell, opens doecho and begins reading commands from that file. It performs the same test on each command, looking for built-in commands, functions, programs, or shell scripts. Each time a shell script is encountered, another copy of the shell is spawned.
I have repeated the running of doecho so you can follow it through the steps described below. The output of doecho is repeated here, with extra spacing and notes.
$ ./doecho                  <-the command typed in shell one
                              launches shell two reading doecho.
doecho says Hello Hollywood <-shell two sets $PLACE and echoes
                              Shell three starts echoplace
echoplace says Hello        <-shell three cannot find $PLACE and
                              echoes a blank
$                           <-shells three and two exit. Back at shell one
As you're looking at a prompt on the screen, you're actually running a top-level shell. If you've just logged on, this will be shell one, where you type the command ./doecho. Shell two is started as a child of shell one. Its job is to read and execute doecho. The doecho script is repeated below.
The first command in doecho creates the shell variable $PLACE and assigns the value "Hollywood" to it. At this point, the $PLACE variable only exists with this assignment inside shell two. The echo command on the next line will print out doecho says Hello Hollywood and move on to the last line. Shell two reads in the line containing ./echoplace and recognizes this as a shell script. Shell two launches shell three as a child process, and shell three begins reading the commands in echoplace.
# doecho sample variable
PLACE=Hollywood
echo "doecho says Hello " $PLACE
./echoplace
The echoplace shell script is repeated below. The only executable line in echoplace is a repeat of the echoed message. However, $PLACE only exists with the value "Hollywood" in shell two. Shell three sees the line to echo echoplace says Hello and the $PLACE variable, and cannot find any value for $PLACE. Shell three creates its own local variable named $PLACE as an empty variable. When it echoes the script, it's empty and prints nothing.
# echoplace echo $PLACE variable
echo "echoplace says Hello " $PLACE
The assignment of "Hollywood" to $PLACE in shell two is only available inside shell two. If you type in a final command in shell one to echo $PLACE at the shell one level, you'll find that $PLACE is also blank in shell one.
$ echo "shell one says Hello " $PLACE
shell one says Hello
$
Thus far, you've only created and used a variable inside of a single shell level. You can, however, publish a shell variable to the environment, thereby creating an environment variable that's available both to the shell that published it and to all child shells started by the publishing shell. Use export in the Bourne and Korn shells.
$ PLACE=Hollywood; export PLACE
$
The Korn shell also has a command that both exports the variable and assigns a value to it.
$ export PLACE=Hollywood
$
The C shell uses a very different syntax for shell and environment variables. Assign a value to a shell variable by using set, then assign an environment variable using setenv. Note that setenv doesn't use the = operator.
> set PLACE=Hollywood
> setenv PLACE Hollywood
Back in the Korn or Bourne shells, if we revisit the doecho script and edit it to export the $PLACE variable, it becomes available in shell two (the publishing shell) and shell three (the child shell).
# doecho sample variable
PLACE=Hollywood; export PLACE
echo "doecho says Hello " $PLACE
./echoplace
When doecho is run, the output is changed. This happens because in shell three $PLACE is found as an environment variable that has been exported from shell two.
$ ./doecho
doecho says Hello Hollywood
echoplace says Hello Hollywood
$
Assigning a value to $PLACE before you run doecho will help you verify its scope. After doecho is complete, echo the value of $PLACE at the shell one level. Notice that doecho in shell two and echoplace in shell three both see $PLACE's value as "Hollywood", but the top-level shell sees the value "Burbank". This is because $PLACE was exported in shell two. The environment variable $PLACE has scope in shell two and shell three, but not in shell one. Shell one creates its own local shell variable named $PLACE that is unaffected by shells two and three.
$ PLACE=Burbank
$ ./doecho
doecho says Hello Hollywood
echoplace says Hello Hollywood
$ echo "shell one says Hello " $PLACE
$ shell one says Hello Burbank
$
Once a shell variable has been exported and become an environment variable, it can be modified by a subshell. The modification affects the environment variable at all levels where the environment variable has scope.
Make some changes to doecho by adding a repeat of the echo line after the return from echoplace.
# doecho sample variable
PLACE=Hollywood
echo "doecho says Hello " $PLACE
./echoplace
echo "doecho says Hello " $PLACE
After it has been echoed, modify echoplace to change the value of $PLACE. Once this is done, echo it again.
# echoplace echo $PLACE variable
echo "echoplace says Hello " $PLACE
PLACE=Pasadena
echo "echoplace says Hello " $PLACE
Retype the previous sequence of commands as shown below. Shell three alters the value of $PLACE, a change that appears in shell three -- and in shell two, even after it returns from echoplace. Once a variable is published to the environment, it's fair game to any shell at or below the publishing level.
$ PLACE=Burbank
$ ./doecho
doecho says Hello Hollywood
echoplace says Hello Hollywood
echoplace says Hello Pasadena
doecho says Hello Pasadena
$ echo "shell one says Hello " $PLACE
$ shell one says Hello Burbank
$
You have seen that the default action of a shell is to spawn a child shell whenever a shell script is encountered on the command line. Such behavior can be suppressed by using the dot command, which is a dot and a space placed before a command.
Execute doecho by starting it with a dot and a space, then echo the value of $PLACE when doecho is complete. In this example, shell one recognizes $PLACE as having been given the value "Pasadena".
$ . ./doecho
doecho says Hello Hollywood
echoplace says Hello Hollywood
echoplace says Hello Pasadena
doecho says Hello Pasadena
$ echo "shell one says Hello " $PLACE
$ shell one says Hello Pasadena
$
Normally, when a shell discovers that the command to execute is a shell script, it would spawn a child shell and that child would read in the script as commands. If the shell script is preceded by a dot and a space, however, the shell stops reading the current script or commands and starts reading in the new script or commands without starting a new subshell.
When you type in . ./doecho, shell one doesn't spawn a child shell, but instead switches gears and begins reading from doecho. The doecho script initializes and exports the $PLACE variable. The export of $PLACE now affects all shells because you exported it at the shell one level.
A dot script is very useful for setting up a temporary environment that you don't want to set up in your .profile. Suppose for instance that you have a specialized task that you do only on certain days, and that you need to set up some special environment variables for it. Place these variables in a file named specvars.
# specvars contains special variables for the
# special task that I do sometimes
WORKDIR=/some/dir
SPECIALVAR="Bandy Legs"
REPETITIONS=52
export WORKDIR SPECIALVAR REPETITIONS
If you execute this variable by simply typing in the name of the specvars file, you won't get the expected effect because a subshell, shell two, is created to execute specvars and the export command exports to shell two and below. Shell one doesn't view these exports as environment variables.
$ specvars
$ echo "WORKDIR IS " $WORKDIR
WORKDIR is
$ 
Using the dot command causes the script to execute as part of shell one; the effect is now correct.
$ . specvars
$ echo "WORKDIR IS " $WORKDIR
WORKDIR is /some/dir
$ 
So there you have some of the ins and outs of shell and environment variables, as well as some ways to get around some of their limitations. If you want to see your current environment variables, type the printenv command; a list of all variables that are available to the current shell, including all child shells, is printed out.

Understanding Unix shells and environment variables, Part 2

Examine and customize your Unix environment

Unix shells come with variables that are used by the shell or related commands. In addition to variables that you create, the shell itself requires or takes advantage of variables that can be set up for it. When you first log in to a Unix system, the /etc/passwd file contains the name of the shell that is to be run for you. This appears in the last field of the password file. To see yours, type cat /etc/passwd and pipe the result through grep looking for your userid. In the example below I have used my id, mjb.

$ cat /etc/passwd|grep mjb
mjb:500:500::/home/mjb:/bin/ksh

 Unix shells and environment variables: Read the whole series! 
In this example, my logon runs the Korn shell. This shell reads and executes any existing file named /etc/profile, which a system administrator has programmed for basic setup actions required for all users. After I execute /etc/profile, I execute $HOME/.profile. This is set up to contain my own environment. Both /etc/profile and $HOME/.profile set environment variables. The Bourne shell works in a similar fashion. The C shell also takes a similar approach, but uses more files. It runs /etc/csh.cshrc, then /etc/csh.login, then an entire raft of files in your home directory, such as ~/.cshrc, ~/.history, ~/.login, and, finally, ~/.cshdirs.
Regardless of the approach, the result is an environment in which the user will run, including environment variables. You can see your environment variables by using printenv or env. The following is a short example of the output.

$ printenv
USERNAME=
HISTSIZE=1000
HOSTNAME=my.system.com
LOGNAME=mjb
MAIL=/var/spool/mail/mjb
TERM=xterm
PATH=/usr/bin:/bin:/usr/local/bin:/usr/bin/X11:/home/mjb/bin
HOME=/home/mjb
SHELL=/bin/ksh
PS1=[\u@\h \W]\$
Shells also use variables that are not part of the environment. For a description of the difference between shell and environment variables, see last month's column.
For example, PS1, listed above as an environment variable, is the prompt displayed on the screen when the shell is waiting for a new command. Another shell variable, PS2, contains the prompt to be used when a command is begun but not completed before Enter is pressed. To see the prompt in use, type the commands below. The first echoes the $PS2 prompt to the screen. Then a new command is started with an opening parenthesis. The user presses Enter immediately and the shell waits for a command and a closing parenthesis. The shell displays the > prompt to indicate that it is waiting for more input. The command is entered and Enter is pressed. Once again, the > prompt is displayed, because the user has not yet closed the open parenthesis. Finally, the user types ) and presses Enter, ending the command.

$ echo $PS2
>
$ (
> cat /etc/passwd|grep mjb
> )
$ 
You can create a more graphic version of this by adding a command to change the $PS2 prompt. In the following example, the value of the $PS2 prompt is changed and the same command sequence is entered. The $PS2 prompt is reset.

$ echo $PS2
>
$ PS2="more please> "
$ (
more please > cat /etc/passwd|grep mjb
more please > )
$ PS2="> "
# echo $PS2
>
$
Why does the PS2 prompt have a value if it is not in the environment? Look at the printenvlisting and you will not see an entry for PS2.

$ printenv
USERNAME=
HISTSIZE=1000
HOSTNAME=my.system.com
LOGNAME=mjb
MAIL=/var/spool/mail/mjb
TERM=xterm
PATH=/usr/bin:/bin:/usr/local/bin:/usr/bin/X11:/home/mjb/bin
HOME=/home/mjb
SHELL=/bin/ksh
PS1=[\u@\h \W]\$
The shell sets up some default shell variables; PS2 is one of them. Other useful shell variables that are set or used in the Korn shell are:

  • _ (underscore) -- When an external command is executed by the shell, this is set in the environment of the new process to the path of the executed command. In interactive use, this parameter is also set in the parent shell to the last word of the previous command.
  • COLUMNS -- The number of columns on the terminal or window.
  • ENV -- If this parameter is found to be set after any profile files are executed, the expanded value is used as a shell startup file. It typically contains function and alias definitions.
  • ERRNO -- Integer value of the shell's errno variable -- this indicates the reason the last system call failed.
  • HISTFILE -- The name of the file used to store history. When assigned, history is loaded from the specified file. Multiple invocations of a shell running on the same machine will share history if their HISTFILE parameters all point to the same file. If HISTFILE isn't set, the default history file is $HOME/.sh_history.
  • HISTSIZE -- The number of commands normally stored in the history file. Default value is 128.
  • IFS -- Internal field separator, used during substitution and by the read command to split values into distinct arguments; normally set to space, tab, and newline.
  • LINENO -- The line number of the function or shell script that is being executed. This variable is useful for debugging shell scripts. Just add an echo $LINENO at various points and you should be able to determine your location within a script.
  • LINES -- Set to the number of lines on the terminal or window.
  • PPID -- The process ID of the shell's parent. A read-only variable.
  • PATH -- A colon-separated list of directories that are searched when seeking commands.
  • PS1 -- The primary prompt for interactive shells.
  • PS2 -- Secondary prompt string; default value is >. Used when more input is needed to complete a command.
  • PWD -- The current working directory. This may be unset or null if shell does not know where it is.
  • RANDOM -- A simple random number generator. Every time RANDOM is referenced, it is assigned the next number in a random number series. The point in the series can be set by assigning a number to RANDOM.
  • REPLY -- Default parameter for the read command if no names are given.
  • SECONDS -- The number of seconds since the shell started or, if the parameter has been assigned an integer value, the number of seconds since the assignment plus the value that was assigned.
  • TMOUT -- If set to a positive integer in an interactive shell, it specifies the maximum number of seconds the shell will wait for input after printing the primary prompt (PS1). If this time is exceeded, the shell exits.
  • TMPDIR -- Where the directory shell temporary files are created. If this parameter is not set, or does not contain the absolute path of a directory, temporary files are created in /tmp.
The C shell uses variables with similar but lowercase names, such as prompt1, prompt2, path, home, and so on.
Other interesting variables are the locale setting variables. These variables are LC_ALL, LC_CTYPE, LC_COLLATE, and LC_MESSAGES. LC_ALL effectively overrides the values for the other three LC variables; you can set them independently by not setting LC_ALL.

  • LC_ALL -- Determines the locale to be used to override any previously set values.
  • LC_COLLATE -- Defines the collating sequence to use when sorting.
  • LC_CTYPE -- Determines the locale for the interpretation of a sequence of bytes.
  • LC_MESSAGES -- Determines the language in which messages should be written.
LC_ALL can be used to change the language for the system. Try the following sequence of commands below to see these in action. The language is changed to French (fr) and grep is invoked with an illegal option -x. The error message appears in French. The LC_ALL is set to Spanish (español, thus es) and the error and error message are repeated. Finally LC_ALL is unset and the error returns in English.

$ export LC_ALL=fr
$ grep -x
Usage: grep [OPTION]...PATRON [FICHIER]
Pour en savoir davantage, faites: 'grep --help'
$ LC_ALL=es
$ grep -x
Modo de empoleo: grep [OPCION]...PATRON [FICHERO]
Pruebe 'grep --help' para mas informacion
$ unset LC_ALL
$ grep -x
Usage: grep [OPTION]...PATTERN [FILE]
Try 'grep --help' for more information.
$
End of article.

The language of shells


Making sense of shell commands

Summary
Working with shells can be difficult, as they require unusual and specific combinations of words and punctuation. This month, Mo Budlong helps you out by explaining some basic commands, such as ls, echo, and man. Also, Mo corrects a problem from May's Unix 101 in a sidebar. (1,300 words)

From the end user's perspective, the shell is the most important program on the Unix system because it is the user's interface to the Unix system kernel. The shell reads and interpreting strings of characters and words.
The shells operate in a simple loop:
  1. Accept a command
  2. Interpret the command
  3. Execute the command
  4. Wait for another command
The shell displays a prompt, notifying the user that it is ready to accept a command. It would be nice if you could speak or type instructions into the computer in some form of natural language.
OK, Hal. Sort out my correspondence, throw out anything
that is too old, and archive the rest.
Unfortunately, the shell recognizes a very limited set of command words, so the user must offer commands in a way that it understands. This means learning to string odd words and punctuation together.
Each shell command consists of a command name, followed, if desired, by command options and arguments. The command name, options, and arguments are separated by blank space.
The shell is one of many programs that the Unix kernel can run for you. When the kernel is running a program, that program is called a process. The kernel can run the same program many times (one shell for each user), and each running copy of the program is a separate process. Because each user runs a separate copy of the shell, each user is running in his or her own process space.
Many basic shell commands are subroutines that are built in to the shell program. The echo command is almost always built in to a shell.
$ echo "Hello, Hal"
Hello Hal
$
Commands not built in to the shell require that the kernel start another process in order to run.
When you execute a command that is not built in to a shell, the shell asks the kernel to create a new subprocess (or child process) to perform the command. The child process exists just long enough to execute the command. The shell waits for the child process to finish before accepting the next command.
The basic form of a Unix command is:
command name [-options] [arguments] 
The square brackets signify parts of the command that may be omitted.
The command name is the name of a built-in command or a separate program you want the shell to execute. The command options, usually indicated by a dash, allow you to alter the behavior of the command. The arguments are the names of files, directories, or programs that the command needs to access.
ls -l /home/mjb
The ls command is usually a separate program rather than a built-in command. The command above will get you a long listing of the contents of the /home/mjb directory. In this example, ls is the command name, -l is an option that tells ls to create a long, detailed output, and /home/mjb is an argument naming the directory that ls is to list.
The Unix shell is case sensitive, and most Unix commands are lower case.
Some of the more popular shells are sh (the Bourne shell), ksh (the Korn shell), csh (the C shell), bash, (the Bourne Again shell), pdksh (the Public Domain Korn shell), and tcsh (the Tiny C shell).
You can frequently identify your shell by typing:
echo $SHELL
Unix recognizes certain special characters as command directives. If you use a special character in a command, make sure you understand what it does. The special characters are / < > ! $ % ^ & * | { } ~ and ;. When naming files and directories on Unix, it is safest to only use numerals, upper and lower case letters, and the period, dash, and underscore characters.
A Unix command line is a sequence of characters in the syntax of the target shell language. Of the characters in a command line, some are known as metacharacters. Metacharacters have a special meaning to the shell. The metacharacters in the Korn shell are:
  • ; -- Separates multiple commands on a command line
  • & -- Causes the preceding command to execute asynchronously (as its own separate process so that the next one does not wait for it to complete)
  • () -- Enclose commands that are to be launched in a separate shell
  • | -- Pipes the output of the command to the left of the pipe to the input of the command on the right of the pipe
  • > -- Redirects output to a file or device
  • >> -- Redirects output to a file or device and appends to it instead of overwriting it
  • < -- Redirects input from a file or device
  • newline -- Ends a command or set of commands
  • space -- Separates command words
  • tab -- Separates command words
Some metacharacters can be used in combinations, such as ||, &&, and >>. With these metacharacters you can define a command-line word, which is a sequence of characters separated by one or more nonquoted metacharacters.
To access the online manuals, use the man command, followed by the name of the command you need help with. For instance, to see the manual for the ls command, enter:
man ls
End of article.
A note to my readers
 
I would like to note a correction to the May edition of Unix 101, in which I said:
"Once a shell variable has been exported and becomes an environment variable, it can be modified by a subshell. The modification affects the environment variable at all levels where the environment variable has scope."
Several sharp eyed readers picked up on this and sent comments ranging from, "Oh, no, you can't" to "Gee, whiz, which shell are you using? It doesn't work for me."
They are right. A subshell cannot modify an environment variable and return it to the parent. It can modify an environment variable and pass it on to a child process, but it cannot return the new value to a higher level. To illustrate this correctly, create the following three script files and grant them execute privileges using chmod a+x script*.

# script1
myvar="Hello" ; export myvar
echo "script1:myvar=" $myvar
./script2
echo "Back from script1 and script2
echo "script1:myvar=" $myvar

# script2
myvar="Goodbye"
echo "script2:myvar=" $myvar
./script3

# script3
echo "script3:myvar=" $myvar
If you run this sequence, the results show that $myvar exists in all three scripts (and, consequently, in all three processes), but modifying it in script2 only affects its value in script3.

$ ./script1
script1:myvar= Hello
script2:myvar= Goodbye
script3:myvar= Goodbye
Back from script 1 and 2
script1:myvar= Hello
$
My apologies to those of you who tried to make the example in the May issue work.

Using cron basics

Utility helps you get your timing right

Summary
Cron allows you to program jobs to be performed at specific times or at steady intervals. This month Mo Budlong explains some cron fundamentals and runs an experiment. (1,500 words)

At one time cron was easy to describe: It involved only one or two files. All you had to do was edit the files and -- voilà! -- cron did the rest. Now cron has become several files and several programs, and at first glance it seems quite complex. Fortunately, someone was clever enough to create a simplified interface along with the new complexity.
Cron is really two separate programs. The cron daemon, usually called cron or crond, is a continually running program that is typically part of the booting-up process.
To check that it's running on your system, use ps and grep to locate the process.
ps -ef|grep cron
root    387      1   0   Jun 29 ?     00:00:00 crond
root  32304  20607   0   00:18 pts/0  00:00:00 grep cron
In the example above, crond is running as process 387. Process 32304 is the grep cron command used to locate crond.
If cron does not appear to be running on your system, check with your system administrator, because a system without cron is unusual.
The crond process wakes up each minute to check a set of cron table files that list tasks and the times when those tasks are to be performed. If any programs need to be run, it runs them and then goes back to sleep. You don't need to concern yourself with the mechanics of the cron daemon other than to know that it exists and that it is constantly polling the cron table files.
The cron table files vary from system to system but usually consist of the following:
  • Any files in /var/spool/cron or /var/spool/cron/crontabs. Those are individual files created by any user using the cron facility. Each file is given the name of the user. You will almost always find a root file in /var spool/cron/root. If the user account named jinx is using cron, you will also find a jinx file as /var/spool/cron/jinx.
    ls -l /var/spool/cron
    -rw-------   1  root    root          3768 Jul 14  23:54  root
    -rw-------   1  root    group          207 Jul 15  22:18  jinx
  • A cron file that may be named /etc/crontab. That is the traditional name of the original cron table file.
  • Any files in the /etc/cron.d directory.
Each cron table file has different functions in the system. As a user, you will be editing or making entries into the /var/spool/cron file for your account.
Another part of cron is the table editor, crontab, which edits the file in /var/spool/cron. The crontab program knows where the files that need to be edited are, which makes things much easier on you.
The crontab utility has three options: -l, -r, and -e. The -l option lists the contents of the current table file for your current userid, the -e option lets you edit the table file, and the -r option removes a table file.
A cron table file is made up of one line per entry. An entry consists of two categories of data: when to run a command and which command to run.
A line contains six fields, unless it begins with a hash mark (#), which is treated as a comment. The six fields, which must be separated by white space (tabs or spaces), are:
  1. Minute of the hour in which to run (0-59)
  2. Hour of the day in which to run (0-23)
  3. Day of the month (0-31)
  4. Month of the year in which to run (1-12)
  5. Day of the week in which to run (0-6) (0=Sunday)
  6. The command to execute
As you can see, the "when to run" fields are the first five in the table. The final field holds the command to run.
An entry in the first five columns can consist of:
  • A number in the specified range
  • A range of numbers in the specified range; for example, 2-10
  • A comma-separated list consisting of individual numbers or ranges of numbers, as in 1,2,3-7,8
  • An asterisk that stands for all valid values
Note that lists and ranges of numbers must not contain spaces or tabs, which are reserved for separating fields.
A sample cron table file might be displayed with the crontab -l command. The following example includes line numbers to clarify the explanation.
1     $ crontab -l
2     # DO NOT EDIT THIS FILE
3     # installed Sat Jul 15
4     #min    hr   day   mon   weekday  command
6     30      *     *     *     *       some_command
7     15,45   1-3   *     *     *       another_command
8     25      1     *     *     0       sunday_job
9     45      3     1     *     *       monthly_report
10    *       15    *     *     *       too_often
11    0       15    *     *     1-5     better_job
$
Lines 2 through 4 contain comments and are ignored. Line 6 runs the command some_command at 30 minutes past the hour. Note that the fields for hour, day, month, and weekday were all left with the asterisk; therefore some_command runs at 30 minutes past the hour, every hour of every day.
Line 7 runs the command another_command at 15 and 45 minutes past the hour for hours 1 through 3, namely, 1:15, 1:45, 2:15, 2:45, 3:15, and 3:45 a.m.
Line 8 specifies that sunday_job is to be run at 1:25 a.m., only on Sundays.
Line 9 runs monthly_report at 3:45 a.m. of the first day of each month.
Line 10 is a typical cron table entry error. The user wants to run a task daily at 3 p.m., but has only entered the hour. The asterisk in the minute column causes the job to run once every minute for each minute from 3:00 p.m. through 3:59 p.m.
Line 11 corrects that error and adds weekdays 1 through 5, limiting the job to 3:00 p.m., Monday through Friday.
Now that you know cron basics, try the following experiment. Cron is usually used to run a script, but it can run any command. If you do not have cron privileges, you will have to follow as best you can, or work with someone who has them.
Use the crontab editor to edit a new crontab entry. In this example I am asking cron to execute something every minute.
$crontab -e
0-59    *    *    *    *    echo `date` "Hello" >>$HOME/junk.txt
$
The sixth field contains the command to echo the output from date (note the reverse quotes around date), followed by "Hello", and also the command to append the result to a file in my home directory, which is named junk.txt.
Close this cron table file. If you have cron privileges and have entered the command correctly, you will receive a receive that the file has been saved.
Use crontab -l to view the file.
$ crontab -l
# DO NOT EDIT THIS FILE
# installed Sat Jul 15
0-59    *    *    *    *    echo `date` "Hello" >>$HOME/junk.txt
$
Change to your home directory, use the touch command to create junk.txt in case it does not exist, and then use tail -f to open the file and display the contents line by line as they are inserted by cron.
$ cd
$ touch junk.txt
$ tail -f junk.txt
Sat Jul 15 15:23:07 PDT Hello
Sat Jul 15 15:24:07 PDT Hello
Sat Jul 15 15:25:07 PDT Hello
Sat Jul 15 15:26:07 PDT Hello
The screen will update once per minute as the information is inserted into junk.txt.
Stop the display by pressing Control-D.
Be sure to clean up the cron table files by using the crontab -e option to open the cron table file and remove the line you just created.
All commands executed by cron should run silently with no output. Because cron runs as a detached job, it has no terminal to write messages to. However, the best-laid plans of mice, men, and programmers are not without deviations from the expected course, and it is entirely possible that a command, script, or job may produce output or, heaven forbid, some actual error messages.
To handle that, cron traps all the output to standard out or to standard error that has not been redirected to a file, as in the example just tested. The trapped output is dropped into a mail file and is sent either to the user who originated the command or to root. Either way, it conveniently traps errors without forcing cron to blow up or abort.

Traveling down the Unix $PATH

Why are some commands executed, and some ./executed?

Summary
Why do some commands need a dot-slash in order to run? In this month's Unix 101 column, Mo Budlong explores the answer to this question, and explains the difference between built-in and executable commands. (1,200 words)

This article is based on a question that came out of July's installment of Unix 101:
Why is it that some commands can simply be executed, while others must be ./executed? In other words, why do some commands need a dot-slash in front of them to run? Rather than giving you a short answer, I am going to explore a couple of things and hope you find them enlightening.
If you create a new shell script, will you be able to run it with the first command below, or will you need to resort to the second?
$ newscript
$ ./newscript
$
Commands in Unix are either builtins or executables. Builtins are part of the shell you are currently running. Examples include echo, read, and export.
Any command that is not built in must be an executable. There are two types of executables: shell script languages, such as sh, ksh, csh, or perl, or compiled executables, such as a program written in C and compiled down to a binary.
Commands created by using an alias in ksh also break down into these two main categories, because the command is translated and then issued as either a builtin or an executable. The following examples create aliases for the builtin echo and the executable grep.
The shell can always locate builtins, because they are built in to the currently executing shell.
$ alias sayit='echo '
$ alias g='grep '
In each case, after alias substitution is completed, the command becomes a builtin or an executable.
You can use the type command to verify the nature of echo.
$ type echo
echo is a shell builtin
$
Now use the type command to check on grep. type will give you the directory that contains the executable grep program.
$ type grep
grep is /bin/grep
$
Whether you enter grep as a command or ask for its location using type, the operating system finds grep by using the $PATH environment variable. If type can find grep, then echoing out the $PATH variable will verify that the path to the directory containing grep is part of the $PATH variable.
$ type grep
grep is /bin/grep
$ echo $PATH
/bin:/usr/bin:/usr/local/bin:/home/mjb/bin
$
The directories listed in $PATH are separated by colons. The above example includes /bin, /usr/bin, /usr/local/bin, and /home/mjb/bin. As an aside, the type command is probably a builtin.
$ type type
type is a shell builtin
$
Another useful command similar to type is whereis, which will usually locate a command and its manual entry.
$ whereis grep
grep: /bin/grep /usr/man/man1/grep.1
$
The shell reads and interprets strings of characters and words typed at the keyboard. Unix shells operate in a simple loop:
  1. Accept a command
  2. Interpret the command
  3. Execute the command
  4. Wait for another command
In step 3, the shell searches for the command to be executed first in the shell itself and then in each of the directories listed in the $PATH. If it can't be found in one of these path directories, an error results.
$ zowie
zowie: command not found
$
It's important to note that the shell does not search the current directory unless that directory happens to be in the $PATH variable. This is important to understand, especially if you came to Unix from an MS-DOS background. MS-DOS uses a PATH variable as well, but it searches the user's the current directory before it searches in any directories in the user's PATH.
Some users have had the foresight to include the current directory in their $PATH variable. This will appear as a single dot, the Unix shorthand for current directory. Note the dot at the end of the $PATH variable below.
$ echo $PATH
/bin:/usr/bin:/usr/local/bin:/home/mjb/bin:.
If you have the dot in your $PATH variable, create a new directory under your home directory, such as $HOME/temp, and change to it.
$ cd $HOME
$ mkdir temp
$cd temp
$
Use the vi editor to create a simple script.
# sayhello
echo "Hello"
Save it and change the mode to executable.
$ chmod a+x sayhello
$
If you have the dot in your $PATH variable, you'll be able to execute the command directly.
$ sayhello
Hello
$
If you don't have a dot in your $PATH variable, the computer will search through your $PATH (anywhere but the current directory) and report failure.
$ sayhello
sayhello: command not found
$
If you type an unadorned command such as sayhello, the computer searches for it. However, if you apply any additional path information to the command, the shell assumes that you are giving an absolute path and only looks where you tell it to. Consequently, ./sayhello locates the command in the current directory.
$ ./sayhello
Hello
$
Obviously, the dot-slash version works whether or not you have a dot in your $PATH variable, because the dot-slash precludes the shell's search for the command.
To get a dot into your $PATH if you don't have one, you need to edit your personal startup profile, usually called .profile and located in your $HOME directory. Look for a line that exports the PATH variable, such as line 5 below. (Line numbers are included here for easy reference, but are not part of the file.) This file already has a line 4 that includes some local additions to the default $PATH.
1.  # .profile
2.  # User specified environment
3.  USERNAME="mjb"
4.  PATH=$PATH:$HOME/bin
5.  export USERNAME PATH
If line 4 did not exist, you'd want to create a line that read:
PATH=$PATH:.
In this case, edit line 4 to read:
PATH=$PATH:$HOME/bin:.
Now, whenever you log in, the dot is added to your search $PATH for commands.
So, the simple rule is: if you want to execute any command in any directory not on your $PATH, including the current directory, you must specify a path to locate the command. This includes a ./ for the current directory.

Security basics, Part 1


Understanding file attribute bits and modes

Summary
In this month's Unix 101, Mo Budlong begins a three-part series on Unix security. In this installment, he explains how to set basic file and directory permissions. (1,500 words)



Security is always an issue in multiuser computing systems. Unix provides a rich set of security options, and this month we begin a three-part security series by exploring some basics.
As a true multiuser, multitasking operating system, Unix has a fairly sophisticated method for setting file and directory permissions.
The chmod command is simple once you've grasped the basics. To understand it, let's start with a small directory listing that can be generated by the ls -l command.

$ ls -l
  drwxrwxr-x    1 mob      wp    2018 Aug 30 23:45 adir
  -rw-rw-r--    1 mob      wp    8755 Aug 30 23:37 picture.gif
  -rwxrwxr-x    1 mob      wp    8525 Sep  4 02:48 command.sh

$
The permissions are indicated by a series of letters on the left-hand side of the listing. The first character indicates the type of the entry. For our purposes, the first character will be either a dash (-) to indicate that the entry is a file or a d to indicate that it's a directory.
After the initial character, a series of rwxs (or the absence of the same) on the left side of the listing indicates the access permissions for that file or entry.
The nine characters after the initial entry-type indicator are broken into three groups, each containing three characters. Each group of three from left to right indicates the permissions given to the owner, group, and others, respectively.
An r indicates that read permission is given to the user who owns the file, group the owner belongs to, or the rest of the world. An r allows a file to be read, and a directory to be listed with ls and related utilities. A w indicates write permission. If this is an entry for a directory, w means that new files can be created within it.
In the following example, the owner, mob, has read and write permissions on picture.gif. The group wp has read permission only, and no other permissions are given.

$ ls -l
  -rw-r-----    1 mob      wp    8755 Aug 30 23:37 picture.gif

$
An x indicates execute permission for executable files, and search permission for directories.
In the following example mob members of the group wp, and all others have search permission for adir.

$ ls -l
  drwxrwxr-x    1 mob      wp    2018 Aug 30 23:45 adir
  -rw-rw-r--    1 mob      wp    8755 Aug 30 23:37 picture.gif
  -rwxrwxr-x    1 mob      wp    8525 Sep  4 02:48 command.sh

$
An x permission on a standard data file has no effect.
To change the permissions on a file, use chmod, followed by the permissions you want to change and the file name. A permission is expressed as a one-character identifier that signifies (u)ser, (g)roup, (o)thers, or (a)ll, followed by +, -, or =, meaning add, remove, or set, respectively. After these characters, add one or more of the following permissions: r, w, or x. The permission strings should look something like the following examples:

u+rw
a-w
o+x
a=rwx
Some more examples are shown below. Line 3 adds write privileges to group, line 6 removes write privileges, and line 9 adds read privileges for others. Line 12 sets privileges for others to write only. In line 14, the r has disappeared and the w has appeared. Line 15 removes read and write privileges for all; note the result at line 17. Line 18 uses ug+rw to add read and write privileges for both user and group.

1  $ ls -l
2    -rw-r-----    1 mob      wp    8755 Aug 30 23:37 picture.gif
3  $ chmod g+w picture.gif
4  $ ls -l
5    -rw-rw----    1 mob      wp    8755 Aug 30 23:37 picture.gif
6  $ chmod u-w picture.gif
7  $ ls -l
8    -r--rw----    1 mob      wp    8755 Aug 30 23:37 picture.gif
9  $ chmod o+r picture.gif
10 $ ls -l
11   -r--rw-r--    1 mob      wp    8755 Aug 30 23:37 picture.gif
12 $ chmod o=w picture.gif
13 $ ls -l
14   -r--rw--w-    1 mob      wp    8755 Aug 30 23:37 picture.gif
15 $ chmod a-rw picture.gif
16 $ ls -l
17   ----------    1 mob      wp    8755 Aug 30 23:37 picture.gif
18 $ chmod ug+rw picture.gif
19 $ ls -l
20   -rw-rw----    1 mob      wp    8755 Aug 30 23:37 picture.gif
$

User and group categories
In a Unix system, a user is a member of a higher-echelon grouping, simply called a group. Some common groups on Unix systems are root, admin, users, and mail.
superuser and root would be members of the root group. Users that have administrative access to backup, restore, and mount operations might be placed in the admin group. Whoever handles the mail system might be in the mail group. The remaining mortals would be in the users group.
A user is given a new group when first created. To see your group, type the following command, using your login where I have mob in the command.

cat /etc/passwd|grep mob
mob:x:537:500::/home/mob/:/bin/ksh
$
The first field, mob, is the login ID. The second field, x, is the encrypted password, which will appear as an x or an unreadable character. The third field is your user ID, the value returned when you type id and press enter. The fourth field, 200, is the group ID. Keep that number written down, and search in your group file to find out which group is 500 and who the members of that group are. In this example, wp is group 500 and is password protected. The members of the wp group are mob and jjk.

cat /etc/group|grep 200
wp:x:500:mob,jjk
$
If you want to see all the groups you're in, repeat the command but grep out your user ID. In this example, mob shows up in the groups wp, accounting, and admin.

cat /etc/group|grep mob
wp:x:500:mob,jjk
accounting:x:512:mob,jan,ded
admin:x:mob,root
$
When you first log in, you're set to the default group specified in your /etc/passwd file.
When you create a new file and then ls -l it, you'll find it's assigned to you as owner and your default group as a group.
This example uses touch to create an empty file, and then displays the directory entry.

$ touch newfile
$ ls -l
  -rw-rw-r--    1 mob      wp    0 Sep 22 23:37 newfile

$
You may change to a new group if you're a member of that group by using newgrp. If the group is password protected, you'll be asked for a password. Once you've changed to the new group, if you create a new file, it will be owned by the new group.

newgrp accounting
$ touch nextfile
$ ls -l
  -rw-rw-r--    1 mob    wp           0 Sep 22 23:37 newfile
  -rw-rw-r--    1 mob    accounting   0 Sep 22 23:37 nextfile

$
As an exercise, you should create a new file with vi and put a couple of lines into it. Close the file and change permissions by adding and subtracting read and write privileges from user, group, and others. Try to edit the file. If you can get your hands on another user login that's not part of your group, try logging in as that person and editing the file.
Here are some general rules. Start with a file that has read and write privileges for all users, -rw-rw-rw-; anyone should be able to edit the file. If you chmod o-rw to -rw-rw----, only you and members of your group will be able to edit it. If you chmod g-rw to -rw-------, only you can edit the file. If you chmod u-w to -r--------, only you can view the document, but you cannot change it or delete it.
Try some similar experiments with a directory. Anyone can read, write (create new files), or search in a directory with a privilege string of drwxrwxrwx. A more typical string for a directory would be drwxrwxr-x, indicating that you and your group have full access, while other users can only read and search. A very secured directory might be set up as dr--------, allowing only the owner to read the directory.

Security basics, Part 2

More advice on file attribute bits and modes

Summary
Could you use a quick refresher course on binary numbers? Need an expert to clarify hexadecimal and octal notation? This month in Unix 101, Mo Budlong continues his three-part series on Unix security with a closer look at file attribute bits and modes. (1,900 words)

While everyone knows that computers are binary, not many people understand the numbering systems that represent the binary numbers stored in computers, or the notational conventions used for displaying this information.
This may be a teensy bit painful, but I need to cover it to further explain setting file modes. If you understand binary numbering, feel free to skip ahead.
Binary numbers
An odometer in an automobile measures mileage by wheels that rotate through the digits 0 to 9. Each time a wheel completes a revolution from 0 through 9 and back to 0, it trips the wheel to its left up one position. When that wheel reaches 9 and is tripped over to 0, it increments the next wheel to its left.
Now imagine that each wheel in your odometer only had the digits 0 and 1. The first wheel would increment from 0 to 1 and back to 0 again. As it rotated back to 0, it would trip the wheel to the left over to 1. Wheel 1 would again spin through 1 and back to 0, and this time it would trip wheel 2 over to 0. This would trip wheel 3 over to 1. The following illustrates the sequence of positions for our wheels. The wheel positions are numbered down the left-hand side so that position 0 is 0000, position 1 is 0001, position 2 is 0010, and so on.
wheel ->   4  3  2  1

position
   0       0  0  0  0
   1       0  0  0  1
   2       0  0  1  0
   3       0  0  1  1
   4       0  1  0  0
   5       0  1  0  1
   6       0  1  1  0
   7       0  1  1  1
   8       1  0  0  0
   9       1  0  0  1
  10       1  0  1  0
  11       1  0  1  1
  12       1  1  0  0
  13       1  1  0  1
  14       1  1  1  0
  15       1  1  1  1
Figure 1. Binary notation
Here are some basics to keep in mind:
  • A byte of computer data is made up of 8 bits, which could be represented by eight wheels.
  • All information in a computer is stored as accumulations of bits and bytes.
  • A byte of data might represent a single character or eight separate flags. Two bytes could be strung together to hold 16 separate flags.
Hexadecimal notation
You could represent the value in a byte by writing out all of the 0s and 1s on each wheel (e.g., 01001101), but that would be cumbersome. Hexadecimal notation is a convenient way of representing the values in a byte with two characters. The characters used are the numbers 0 through 9 and the letters A through F. Figure 2 illustrates how a single hexadecimal digit can represent 4 bits.
wheel ->   4  3  2  1

position                  Hex
   0       0  0  0  0      0
   1       0  0  0  1      1
   2       0  0  1  0      2
   3       0  0  1  1      3
   4       0  1  0  0      4
   5       0  1  0  1      5
   6       0  1  1  0      6
   7       0  1  1  1      7
   8       1  0  0  0      8
   9       1  0  0  1      9
  10       1  0  1  0      A
  11       1  0  1  1      B
  12       1  1  0  0      C
  13       1  1  0  1      D
  14       1  1  1  0      E
  15       1  1  1  1      F
Figure 2. Binary numbers and their hexadecimal equivalents
The 8 bits of a byte are frequently represented by grouping the bits into two collections of 4 bits each, then representing each of these with a hexadecimal digit. In our case, the number 00101101 becomes 0010 1101, and that, in turn, becomes 2D.
Octal notation
Hexadecimal notation represents bits in groups of four. Octal notation slices a byte and represents it in groups of three, as shown in Figure 3.
wheel ->   3  2  1

position               Oct
   0       0  0  0      0
   1       0  0  1      1
   2       0  1  0      2
   3       0  1  1      3
   4       1  0  0      4
   5       1  0  1      5
   6       1  1  0      6
   7       1  1  1      7
Figure 3. Octal notation
Using this technique, 00111101 becomes 00 111 101 which, in turn, becomes 075 in octal notation. A byte only has 8 bits, so the highest 2 bits are presented as if they had a leading 0, and 10111101 becomes 010 111 101, or 275 in octal. In fact, longer strings of bits can be represented by longer hexadecimal or octal notation. The 16 bits of 2 bytes, 00101011 10010110, become 0010 1011 1001 0110, which can be represented as 2B96 in hexadecimal. If you divide the number into groups of 3 bits and add two extra 0s to the beginning, you get 000 010 101 110 010 110, or 025626 in octal.
The mode bits
From last month's article, you know that the ls -l mode bits are displayed (from left to right) as read, write, and execute for owner; read, write, and execute for group; and read, write, and execute for other.
-rwxrwxr-x    1 mob      wp    2018 Aug 30 23:45 afile
These nine flags are actually saved as 9 mode bits (a byte plus part of another), and the mode bits can be represented by the octal notation for that bit pattern.
The above flags represent a bit pattern of 111111101 or 111 111 101, and can be expressed as 775. If you wanted to set those exact permissions for the file, you would use the following command:
$ chmod 775 afile
$ ls -l
-rwxrwxr-x    1 mob      wp    2018 Aug 30 23:45 afile
Figure 4 shows several examples of directly setting mode bits using octal notation. The last example makes the file read only.
$ chmod 775 afile
$ ls -l
-rwxrwxr-x    1 mob      wp    2018 Aug 30 23:45 afile
$ chmod 770 afile
$ ls -l
-rwxrwx---    1 mob      wp    2018 Aug 30 23:45 afile
$ chmod 750 afile
$ ls -l
-rwxr-x---    1 mob      wp    2018 Aug 30 23:45 afile
$ chmod 740 afile
$ ls -l
-rwxr-----    1 mob      wp    2018 Aug 30 23:45 afile
$ chmod 444 afile
$ ls -l
-r--r--r--    1 mob      wp    2018 Aug 30 23:45 afile
Figure 4. Setting mode bits using octal notation
In fact, there are 3 more bits available for controlling the mode of files and directories, but these display in different ways. Two of the bits apply to files, and one to directories.
The first bit controls the set user ID property of an executable program or shell script. When a program or script has this bit set and is executed, the script assumes the privileges of the owner of the script. The purpose of this mode would be to provide something like a backup script. The script owner would be root and root would have the privileges needed to back up all files, but the script could be executed by any backup operator who wouldn't need to be given root privileges in order to run the backup.
The second bit controls the same feature for the group. The program or script acquires the privileges of the group that owns the file.
The third bit controls the behavior of directories and is popularly called the stick bit. When this bit is set on a directory, the only people who can delete or rename files from that directory are root and the owner of the directory, regardless of any other permission a user's been granted. This is frequently used on temporary and work directories where many users need to be able to write to the directory, but where no one should be allowed to rename or delete anyone else's files.
These 3 bits are added at the front of the 9-bit bit pattern for access permission, creating a 12-bit pattern. In Figure 5, the first command prevents anyone but root or mob (the directory owner) from deleting or renaming any files in adir. Note the t in the final position of the permission string. The second command allows anyone to run the backup script, but when the script runs it has the privileges of root. Note the s:
$ chmod 1777 adir
$ ls -l
drwxrwxrwt    1 mob      wp    2018 Aug 30 23:45 adir
$ chmod 4111 backup
$ ls -l
---s--x--x    1 root     wp    2018 Aug 30 23:45 backup
Figure 5. Using a full 12-bit pattern for permissions
There is a security feature built in to the set user ID bit and set group ID bits that causes the bit to be reset if the file is written or renamed by anyone other than the superuser. This prevents someone from editing a script that has extra privileges, because root or someone with more privilege owns the script and the set user ID bit is set.
The character representation of these extra 3 bits worth of permission/behavior is handled by cramming their values into the existing permission string with different letters.
The normal state of the owner execute flag is either x or - (dash). If the owner has execute permission and the set user ID bit is on, the x becomes an s. If the owner does not have execute permission but the set user ID bit is on, the - becomes an S, as in Figure 6.
$ chmod 100 backup
$ ls -l
---x------    1 root     wp    2018 Aug 30 23:45 backup
$ chmod 4100 backup
$ ls -l
---s------    1 root     wp    2018 Aug 30 23:45 backup
$ chmod 4000 backup
$ ls -l
---S------    1 root     wp    2018 Aug 30 23:45 backup
Figure 6. The difference between s and S for a file owner
The normal state of the group execute flag is either x or -. If the group has execute permission and the set group ID bit is on, the x becomes an s. If the group does not have execute permission but the set group ID bit is on, the - becomes an S, as in Figure 7.
$ chmod 010 backup
$ ls -l
------x---    1 root     wp    2018 Aug 30 23:45 backup
$ chmod 2010 backup
$ ls -l
------s---    1 root     wp    2018 Aug 30 23:45 backup
$ chmod 2000 backup
$ ls -l
------S---    1 root     wp    2018 Aug 30 23:45 backup
Figure 7. The difference between s and S for a file owner's group
The normal state of the execute flag for others for a directory is either x, indicating that others can search the directory, or -, indicating that they cannot. If others have search permission and the sticky bit is on, the x becomes a t. If others don't have search permission but the sticky bit is on, the - becomes a T.
$ chmod 001 adir
$ ls -l
d--------x    1 root     wp    2018 Aug 30 23:45 adir
$ chmod 1001 adir
$ ls -l
d--------t    1 root     wp    2018 Aug 30 23:45 adir
$ chmod 1000 adir
$ ls -l
d--------T    1 root     wp    2018 Aug 30 23:45 adir
Figure 8. The difference between t and T
Now you have two ways to set up the modes for a file and directory, and three extra security permissions to control the access level of executable programs and protect files within a directory from being deleted or renamed.