
Common shell script mistakes (2008) - pmoriarty
http://www.pixelbeat.org/programming/shell_script_mistakes.html
======
illicium
Shout out to Shellcheck
([http://www.shellcheck.net/](http://www.shellcheck.net/)), a linter for shell
scripts

~~~
eridal
If you're on Sublime, this works like a charm!

[https://github.com/SublimeLinter/SublimeLinter-
shellcheck](https://github.com/SublimeLinter/SublimeLinter-shellcheck)

~~~
jasonm23
If you're on Vim or Emacs you already (maybe) know you have integration with
shellcheck

Emacs -
[https://github.com/koalaman/shellcheck](https://github.com/koalaman/shellcheck)

Vim - part of syntactic -
[https://github.com/scrooloose/syntastic/blob/master/syntax_c...](https://github.com/scrooloose/syntastic/blob/master/syntax_checkers/sh/shellcheck.vim)

------
pvg
The google shell style guide linked now lives here and is probably an even
better starting point than this article:

[https://google.github.io/styleguide/shell.xml](https://google.github.io/styleguide/shell.xml)

~~~
pmarin
The google shell style uses bash which is not portable, one of the main point
of the OP article.

~~~
nailer
Bash is the default shell on the most popular server nix (Linux) and the most
desktop nix (OS X, though it's always a slightly old version).

If you use OpenBSD and hate ports, then maybe ignore all the good things bash
brings and write everything in Bourne shell. If not, then write bash. Just
fire it up with '/usr/bin/env bash' to label it correctly.

~~~
JdeBP
> _Bash is the default shell_

Not for nigh on a decade, now.

* [https://wiki.ubuntu.com/DashAsBinSh](https://wiki.ubuntu.com/DashAsBinSh)

* [https://wiki.ubuntu.com/DashAsBinSh/Spec](https://wiki.ubuntu.com/DashAsBinSh/Spec)

* [https://lists.debian.org/debian-release/2007/07/msg00027.htm...](https://lists.debian.org/debian-release/2007/07/msg00027.html)

* [https://wiki.debian.org/DashAsBinSh](https://wiki.debian.org/DashAsBinSh)

~~~
nailer
Bourne shell being replaced as /bin/sh by dash doesn't change anything: Linux
and OSX don't use /bin/sh as the default shell.

~~~
JdeBP
> _Linux and OSX don 't use /bin/sh as the default shell._

You didn't even read the first sentence of the first hyperlinked page.

~~~
the_why_of_y
I believe xe meant "default login shell". Also, "Bourne Again shell", while
we're complaining about missing words :-)

~~~
JdeBP
Xe would have been a bit daft to mean that, given the headline topic as well
as that the conversation so far has dealt in shell scripts and shell script
interpreters, to which the choice of login shell is almost wholly irrelevant.

------
stewartbutler
So many edge cases. I prefer to just use perl if I am doing anything more
complicated than making a wrapper for some other executable. Harder to screw
up, easier to test, same expressiveness, better portability (works with all
shells and with windows), plus you get regular expressions for free.

Python is a good alternative as well.

~~~
s_kilk
Perl is brutally under-rated as a scripting tool these days. I suppose it's
passed into software orthodoxy by this point, the notion that perl is a
uniformly horrific tool akin to nuclear waste.

~~~
naivepiano
I think that this is because people are finding Python equally well suited for
such tasks and more modern. Perl (easy to) obscure syntax is not helping
either.

~~~
jerf
If you're doing a really grungy, grotty text mangling task in a UNIX pipeline,
suddenly all those crazy Perl features start making sense. The reason why Perl
is arguably not the best choice for other tasks, such as creating large
systems, is precisely that it's actually got many distinct advantages over
Python for that sort of task. It's also really good for "I have a UNIX
pipeline but this one chunk of it needs a database and this other chunk needs
a JSON file and a few other things that shell just can't do".

For instance, for the first time in 15+ years, I recently had a reason to use
Perl's half-deprecated, off-handedly-included form printing support, and, you
know what? It's _really nice_ if that's what you want. Many dozens of lines of
code replaced by what is basically a picture of what I want.

Python is a great language, but every once in a while "explicit is better than
implicit", which is generally a philosophy I agree with wholeheartedly,
results in a _lot_ more "explicit", in the form of "code I had to write", than
you might be looking for. (Or, alternatively, you start writing Python that is
a lot more implicit, which isn't exactly a huge victory for Python here.)

Nowadays I'd also suggest that if you find yourself having a lot of those
tasks, you need to ask some questions about how you're storing your data. But
pretty much everybody ends up with some of these somewhere; it's hard to
eliminate them, and not really worth it (due to diminishing returns).

------
soyiuz
Great post. I disagree on using the concise form of the if statement, however.

    
    
      if [ "$var" = "find" ]; then
        echo "found"
      fi
    

Is far more readable than its equivalent

    
    
      [ "$var" = "find" ] && echo "found"
    

I understand the upside of readability. What does concision get me?

~~~
pwd_mkdb
concision gets me less code to read. that's how i define "readability". but if
forum commments are any indication, i know my preferences do not follow the
norm.

most programmers seems to prefer verbosity.

however in my case verbosity slows me down.

~~~
wodenokoto
By that logic you find minified javascript easier to read too.

~~~
nikita2206
This is not @pwd_mkdb's logic though, it's you who brought this to the
extreme.

~~~
coldtea
Not really. @pwd_mkdb said he defines readability as "less code to read".

If that's the only criterion, then the parent just followed the logic. Or
let's put it another way, if the definition of readability can't survive
taking it to the extreme, it is flawed.

~~~
cgriswald
> If that's the only criterion, then the parent just followed the logic.

wodenokoto offered up some _reductio ad absurdum_ and you are calling it
logic.

It was not the only criterion. It was the only criterion that was explicit.
There are implied criterion and most of us understand what pwd_mkdb means even
if we don't necessarily agree with him.

~~~
coldtea
> _It was the only criterion that was explicit._

Well, even in itself, it is wrong.

It's not just that other criteria apply too -- it's that it alone needs
several caveats, as succinctness is quite orthogonal to readability (e.g.
sometimes even needless boilerplate syntax that the compiler could infer by
itself, make for better readability when present).

~~~
cgriswald
> Well, even in itself, it is wrong.

I am not in disagreement with you about whether it is wrong, I am in
disagreement with you about _why_ it is wrong.

The counter argument was that Y is wrong and X is Y, therefore X is wrong.
While this is a valid argument, it is not sound, because X is not Y.

> ... succinctness is quite orthogonal to readability ...

There is a relationship between succinctness and readability. The relationship
is definitely not directly proportional as the pwd_mkdb's post could be read
to imply, but to say there is no relationship between the two is flatly
absurd.

------
gizi
Bash is the love of my life! I have been working for years on this problem now
(not full-time of course), gradually moving in the direction of finally being
able to challenge this:

"Inappropriate use

shell is the main domain specific language designed to manipulate the UNIX
abstractions for data and logic, i.e. files and processes. ...
Correspondingly, please be wary of writing scripts that deviate from these
abstractions, and have significant data manipulation in the shell process
itself. While flexible, shell is not designed as a general purpose language
and becomes unwieldly when ... "

Another person has actually solved the most important show stopper already:
[http://ctypes.sh](http://ctypes.sh).

What now remains to be solved, are a few minor, additional details, and then
simply writing a good manual of how to very successfully use bash as a
general-purpose language.

My personal belief is that everything that you can do in other scripting
languages, you can also do in Bash, only better.

~~~
majestik
Ok, I'll bite.

>> My personal belief is that everything that you can do in other scripting
languages, you can also do in Bash, only better.

1) Native JSON, XML

2) Classes, namespacing, objects

3) Multiprocessing, multithreading

4) Performance

5) Package management

6) Portability

7) Documentation

8) Runtime debugging (!set -x)

I'm too tired to continue.

~~~
int0x80
>3) Multiprocessing

IMO shell makes it very easy to work with multiple process (&). It's built in
and natural.

>4) Performance

If you are carefull and know what you're doing, you can achive very good
performance with the shell. Usually, better performance is achived processing
less data, ie being inteligent. Rarely depends on the language (unless you
care about cycle level performance, then yes :).

>6) Portability

I claim that it's way easier to depend on sh being on a (UNIX) system than
$SCRIPTING_LANG.

>7) Documentation

?? You can mess up documentation in any language.

~~~
jerf
Shell makes it easy to spawn multiple processes. It makes it reasonably easy
to read those processes' standard out or standard error, though it's not that
much fun to try to do both at the same time while keeping them distinct. [1]

It pretty much doesn't do anything else that you might want to do with
multiple processes, though, and it tends to encourage multiple processes to
communicate via text which is a problematic limitation that one often finds
oneself "working around".

Shell is really powerful, _but_ it hits a certain limit of what kind of tasks
it can do and it hits that limit _hard_ , and that's why when one imagines
orchestrating many processes on a machine to do some task, to say nothing of
orchestrating many processes on many machines, you don't see solutions based
on shell, and indeed the very idea is laughable. Shell is best used by making
sure it stays firmly restricted to the domain it shines in and not so much as
trying to dip a toe into the spaces where it is not.

[1]: Note "not much fun" != "can't". Shell is fundamentally written around the
idea that a process has one stream STDOUT that may go to other processes, and
one stream STDERR which is generally intended to go to the console (or other
user output like a log) no matter how complicated the pipeline. While you
_can_ get both streams and do things to them, you're starting to fight shell,
which really wants to create pipelines with one "through" path with no
branches out.

~~~
int0x80
I think with the shell you have to adapt your abstractions to the "unix-way".
For example, a queue to process will be a directory with N files, and each
file can be processed in pararell by just something like "for f in dir/*; do
process.sh "$f" & done;" but yeah ... it has limitations like everything.

------
kps
The big missing bashism in the list is the use of &> or >& to redirect both
standard output and standard error.

In a POSIX conforming shell,

    
    
        &> word
    

is the same as

    
    
        & > word
    

since '&>' is not a distinct token; it puts the command in the background and
redirects standard output.

The more commonly seen >& will bite you even in bash. What does this do?

    
    
        >& $FILENAME
    

Answer: in bash, it _depends on the spelling of the file name_.

Note that bash accepts both even in its so-called ‘POSIX compatible’ mode.

------
paulddraper
> please be wary of writing scripts that deviate from these abstractions, and
> have significant data manipulation in the shell process itself.

I appreciate he mentions this first. Shell scripting is excellent in its
domain, just as SQL is great in its.

You would write an application in SQL and you shouldn't write one in shell.

------
Too
> Just test the string directly like [ "$var" ] && echo "var not empty"

Will this work if you've activated _set -u_ , fail on unset variables? Will
the !-z variant do?

~~~
brbsix
If `set -u` is enabled, the aforementioned test will fail on unset variables.
To avoid this, you can "declare" `var` at the beginning of the script, e.g.
`var=''`. Also instead of `[ ! -z "$var" ]` I prefer `[ -n "$var" ]`, which is
the same as `[ "$var" ]` but more explicit.

------
bootload
_" Writing shell scripts leaves a lot of room to make mistakes, in ways that
will cause your scripts to break on certain input, or (if some input is
untrusted) open up security vulnerabilities. Here are some tips on how to make
your shell scripts safer.

Don't"_ [0]

[0] MIT, writing safe shellscripts ~ [https://sipb.mit.edu/doc/safe-
shell/](https://sipb.mit.edu/doc/safe-shell/)

------
zAy0LfpBZLC8mAC
That temp file example is a vulnerable to symlink attacks.

------
olalonde
Is there a modern "JavaScript: The Good Parts" for bash? There are so many
ways to do things and it's often hard to tell which is preferable.

~~~
arthulia
Not sure if this is what you're looking for, but it's where I tend to go when
writing something.

[http://mywiki.wooledge.org/BashGuide](http://mywiki.wooledge.org/BashGuide)

~~~
stirner
I'm a big fan of this guide. I wish every language had such a clear best
practices list. See also
[http://mywiki.wooledge.org/BashPitfalls](http://mywiki.wooledge.org/BashPitfalls)

------
ourmandave
Written in 2008.

~~~
gberger
Shell hasn't changed much, if at all, since, has it?

~~~
pwd_mkdb
nope. reminds me of comments where someone is purporting to be able to assess
the quality of software based only on looking to see when the last changes to
the source code were made.

~~~
eximius
I'm not saying that is a valid strategy, but man I see more bad old code than
new code.

