Consultix Shell Quoting Guidelines

Introduction

One routinely sees in comp.unix.shell people asking for help with defective shell scripts that suffer from incorrect quoting! So I hereby provide to all interested parties my Shell Quoting Guidelines, derived from the personal anguish and insights of my 23 years of UNIX experience!

The document that follows is divided into three parts:
1) Shell Quoting Guidelines
2) Detailed examples of the Shell Quoting Guidelines in Action
3) Restrictions Regarding Distribution and Use

Quoting is a difficult subject, and I purposely steered clear of certain complex situations in the detailed examples shown below. Instead, I have chosen to concentrate on the situations users are most likely to encounter, to maximize the likelihood of getting the message across and minimize the likelihood of the inadvertent bamboozlement of the intended beneficiary. (I intend to add examples featuring more complex quoting situations in a subsequent version.)

Despite the omission of really complex cases, the Guidelines have evolved with due heed being paid to their consideration, so they are in fact intended to prescribe the correct approach in all cases. "Your mileage may vary", however, so let me know if you find cases where they don't work so well.

You might want to print out the Guidelines page and keep it near your workstation for easy access. I sure do!

Intended Audience

Cheers,

Tim Maher
CONSULTIX
tim(AT)teachmeunix.com
http://www.teachmeunix.com

WHEN SHELL PROCESSING OF MATERIAL ON COMMAND LINE *IS* DESIRED

  1.  for File Name Generation requests do not use any quoting
  2.  for Variable or Command Substitution Requests use double quotes,
	unless there is a contra-indication:
        a.  File Name Generation is desired on result of substitution
        b.  treating result as a single word is not desired
        c.  preserving resulting NULL string is not desired

WHEN SHELL PROCESSING IS *NOT* DESIRED

  3.  use backslash to quote a single character, or single quotes to
      quote multiple characters,
        unless there is a contra-indication:
        a.  characters are not special to the shell

AFTER APPLYING THE QUIDELINES

  4.  seek to simplify resulting command by consolidating quoting techniques


Glossary:
        FILE-NAME-GENERATION: the "wild-card" notation of the shells
        that allows * to be replaced by names of all (non-hidden) files,
        A?C to be replaced by names of all files that consist of three
        letters beginning with A and ending with C, etc.

        SHELL PROCESSING: refers to the shell's examination of 
        command line text and possible alteration of it through
        variable substitution, command substitution, etc.

NOTES: In the examples below, the $ prompt indicates the Bourne/Korn shell, and % the C-shell. DQs stands for Double Quotes, and SQs Single Quotes.

QUOTING GUIDELINES: EXAMPLES

1. grep 'ab?' *

A literal command name, in this case grep, will generally not need any quoting, because it will not ordinarily contain any special characters. So that was left unquoted, in accordance with Guideline 3a. The argument ab? is enclosed in single quotes, to prevent any shell processing, in accordance with Guideline 3. The filename generation request, *, is left unquoted, in accordance with Guideline #1.

2a. % set files=\*; echo "**$LOGNAME's Files**" $files

csh case:

The backslash is used to prevent the shell from interpreting the * as an FNG request, in accordance with Guideline #3; its use permits assignment of the * itself to the variable called "files."

Application of the Guidelines to the echo command produces the following result: echo '**'"$LOGNAME"\''s Files**' $files

The SQs around the two initial asterisks are dictated by Guideline #3. Guideline #2 prescribes the DQs around the variable substitution request $LOGNAME, Guideline #3 prescribes the backslash before the next SQ (used as an apostrophe), as well as the paired SQs around the material ending with the asterisk.

The quoting situation that results is unnecessarily complex, however, and consideration of Guideline #4 allows simplification to that shown above, making use of the fact that DQs all by themselves prevent expansion of *, disable recognition of the SQ as a special character, and permit expansion of the variable substitution request.

All that is now left to consider is the reference to $files at the end of the line. According to Guideline #2, one should generally use DQs around a variable substitution request, but this one must be let unquoted to allow expansion of the * that results from the substitution (contra-indication 2a).

2b. $ files=\*; echo "**$LOGNAME's Files**" $files

sh/ksh case:

For a detailed explanation of the issues involved in quoting this line, see the C-shell case covered above.

The quoting shown here is identical to that shown for the C-shell version above. However, if we want to get picky, we could point out that the backslash before the asterisk is not required to prevent expansion of * in the assignment to the variable "files", as it was in the C-shell case, because in the Bourne/Korn shells FNG processing is not performed for assignment statements anyway. However, application of Guideline #3 by someone who does not know this would dictate the use of the backslash here, and it doesn't hurt, so what the heck!

3a. % set pattern='two words'; grep "$pattern" $files

3b. $ pattern='two words'; grep "$pattern" $files

Guideline #3 says use SQs around the material being assigned to the variable called "pattern" to prevent shell processing. This permits the assignment of multiple words to the variable by removing the word-delimiter meaning of the embedded whitespace character.

$pattern as a variable substitution request gets DQs, according to Guideline #2, because none of the contra-indications applies.

$files, on the other hand, must not be treated as a "single word", because that would cause grep to receive all filenames resulting from the variable substitution as a single composite filename (contra-indication #2b). Therefore, contra-indication #2b applies, and quotes are not used.

If FNG is desired on the result of variable substitution for $files (which might contain something like *), then contra-indication #2a would also apply, again prescribing a lack of quoting.

4. grep "`cat pfile`" `find "$START" -type f -print`

grep needs to receive its pattern as a single argument, so DQs, prescribed by Guideline #2, are required here to render the result of the command substitution request involving the cat command a single word. In contrast, individual filenames must be presented as separate arguments (for the same reason discussed above), so the command substitution request involving the find command must be left unquoted (contra-indication #2b). The DQs around $START are prescribed by Guideline #2, and they will ensure an error message from "find" with sh and ksh if that variable happens to be unset (the csh, on the other hand, would issue that message itself, and terminate execution if working from a script).

5a. $ grep "`grep \"$pattern\" file`" other_file

Variation 5a on the preceding example shows a case in which nested DQs are prescribed by Guideline #2, because we have a variable substitution request nested within a command substitution request. This is easily handled by the Bourne and Korn shells by quoting the inner DQs with a backslash (which is not prescribed by our Guidelines, due to their focus on non-nested commands).

A direct C-shell counterpart to this command is difficult to identify, due to the inability of the programmer to disable the special meaning of a DQ within a DQ-quoted string in that shell.

6a. $ if test "$foo" != bar; then command1; fi

6b. % if ( "$foo" != bar ) command1

Because != is a binary operator, it is imperative that a null string be preserved as an argument if the $foo request should happen to produce one; otherwise, "test" generates a syntax error and a shell script would terminate! Guideline #2 prescribes DQs there, and that's exactly what's needed to prevent the shell from discarding the possible NULL-string result.

In the C-shell case, the same problem would arise if foo happened to be set to a null string, so once again the Guidelines prescribe the correct quoting technique.

We don't want the shell to tamper with bar, but it is not composed of any special characters anyway, so it is left unquoted according to Guideline #3a.

(NOTE: A "need for symmetry" makes some people feel uncomfortable unless both operands are encased in some kind of quotes; there's no harm in enclosing bar in SQs, so feel free to do so if you wish.)

7a. grep "$pattern" $1

7b. grep "$pattern" $*

As with several preceding examples, the variable substitution request $pattern gets DQs, according to Guideline #2.

The second variable substitution request in each command provides a more interesting scenario. Specifically, in shell scripts that accept invocation options (-a, -b, etc.) initial option arguments are often shifted away so that nothing but filenames will be produced by later references to the parameter list variables. For any command that will read stdin if not provided with a filename argument, it would be undesirable to quote such variable references (e.g., "$1"), because if no filename were provided we'd like to have the command look to stdin for input. (This allows pipeline usage of a script that can also accept filename arguments.) So the cases above illustrate variable substitution requests for which contra-indication #2c applies, which is why they have been left unquoted.

In case 7b, contra-indication #2b also applies to the second variable substitution request, because if we applied DQs we'd present multiple filenames as a single argument, and grep would fail. (Another possibility would be usage of "$@" in the Bourne/Korn shells, but many older Bourne shells don't handle that correctly, few programmers know about it, and we've decided to defer discussion of that option to later versions of this document.)

Tim Maher, Ph.D.
CONSULTIX
INTERNET: tim(AT)teachmeunix.com

RESTRICTIONS GOVERNING DISTRIBUTION AND USE

With respect to the distribution and use of the "CONSULTIX Shell Quoting Guidelines" (hereinafter referred to as "Material"), the author and copyright holder, Timothy F. Maher, requires the following conditions to be met:

- this Material shall not be sold without the author's written permission
- this notice shall accompany the Material in all distributions
- any changes to the Material shall be identified in a conspicuous manner in all distributions

There is no warranty of merchantability nor any warranty of fitness for a particular purpose nor any other warranty, either express or implied, as to the accuracy of the herewith published Material, or as to its suitability for any particular purpose.

Accordingly, the author assumes no responsibility for the use of this Material by the recipient. Furthermore, the author assumes no obligation to furnish any assistance of any kind whatsoever, or to furnish any additional information or documentation.

Having gotten the legal mumbo-jumbo out of the way, I'd like to state that these Shell Quoting Guidelines are correct, to the best of my knowledge, and that they have been used extensively in the daily operations of UNIX systems to the satisfaction of many end-users and programmers.

I think you'll find them useful, and wish you the best of luck with them! I'd be interested to hear your comments, suggestions, or (knock wood) bug reports.

Tim Maher, CEO
Consultix
tim(AT)teachmeunix.com
http://www.teachmeunix.com