
Bash Pitfalls - aaron-santos
https://mywiki.wooledge.org/BashPitfalls
======
nikisweeting
I consider shellcheck absolutely essential if you're writing even a single
line of Bash. I also start all my scripts with this "unofficial bash strict
mode" and DIR= shortcut:

    
    
        #!/usr/bin/env bash
        
        ### Bash Environment Setup
        # http://redsymbol.net/articles/unofficial-bash-strict-mode/
        # https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html
        # set -o xtrace
        set -o errexit
        set -o errtrace
        set -o nounset
        set -o pipefail
        IFS=$'\n'
    
        DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
    

I have more tips/tricks here: [https://github.com/pirate/bash-
utils/blob/master/util/base.s...](https://github.com/pirate/bash-
utils/blob/master/util/base.sh)

~~~
abathur
I've had a thought percolating (for many months?) but I haven't tried to
phrase it and I'm leery it may sound condescending (this is also more of a
public address than direct response) ...

Shellcheck tends to steer us away from "weird" parts of Bash/shell and will
happily suggest changing code that was correct as written. There are good
reasons for this (and I still think most scripts/projects should use
Shellcheck), but Bash is a weird language, and there's a lot of
power/potential in some of the weird stuff Shellcheck walls off.

If I had to nail down some heuristics for working with it...:

\- do lean heavily on Shellcheck for writing new code if you're unfamiliar
with the language or are only writing it under duress

\- don't implement its suggestions in bulk (the time you save on iterating can
easily be blown later trying to debug subtle behavior changes)

\- don't apply them to code you don't understand (if you have the time and
interest to understand it but find yourself stuck on some un-searchable
syntax, explainshell may help)

\- don't adopt them without careful testing

\- do take Shellcheck warnings about code that is correct-as-written as a hint
to leave a comment explaining what's going on

\- do leave Shellcheck off (or only use it intermittently) if you're trying to
explore, play, or otherwise learn the language

~~~
cle
For me this is way too much to invest in Bash. Any non-trivial utilities
should be written in a real programming language when possible, especially
when they are used in a context where they are likely to accrete complexity
over time. I’m sure there are good use cases for complex Bash programs, but we
should be reaching for something with less footguns by default.

(Shellcheck is an amazing utility and I use it for all Bash that I write.)

~~~
abathur
What is? Did I suggest anyone write complex Bash programs?

Shellcheck is amazing. As I suggested in the previous post, I use it for most
of my own shell scripts/projects, especially stuff I intend to release. But,
because shell and Bash are weird and full of pitfalls, I think it's worth
making sure people know you can't just Shellcheck-and-ship.

~~~
oblio
I think his nitpick was you promoting weird bash tricks. They're neat to use
but usually a pain for someone else to maintain, as a general rule, don't use
them, don't try to be clever unless you're 100% sure that it's code you'll be
the only one to see (which can rarely be guaranteed).

------
zests
Bash is one of my favorite programming languages. Not for production. Writing
significant production Bash that others have to use and maintain is more of a
struggle than it is worth.

But for personal use? Hitting APIs, organizing data, writing to files... Bash
has been enjoyable because it requires much more creativity than almost any
other language. At some point it's just you and the coreutils against the
world.

~~~
bor100003
I have the same feeling. In most cases you do the job without any external
libs which is rarely the case with other programming languages.

------
Ambol
These type of things is why I always use a real programming language rather
than shell scripting. Can't wrap my head around all the different pitfalls of
treating random string outputs as input for other commands without first
validating and parsing them into an actual data structure.

~~~
renewiltord
What frameworks/libraries do you use? I find that the reason I prefer using a
shell script is:

* ergonomic integration with the shell

* easy piping

* easy input/output redirection

Maybe Ruby comes close with the backticks or `system` but it still isn't as
nice as a shell IMHO

~~~
GuB-42
For me, it would be Perl.

Perl is influenced by shell scripting and there are several easy ways to run
commands, build pipes, redirect and other things shells are good at, like
testing and listing files.

And as ugly as Perl looks, it has much less pitfalls than shell scripts.

~~~
renewiltord
I wrote some Perl code once and it looks like you need more than a passing
familiarity with the language to create code that is not just write-once.

~~~
GuB-42
With proper development practices, you can write clean Perl, even if you don't
know much about the language. It has the same constructs as most other
procedural languages and you can apply the same principles. Object orientation
is a bit tricky though.

The thing is, Perl won't help you with discipline. If you want write-only
code, Perl will compile it, no problem.

For that reason, it is the language of choice for one liners and throwaway
scripts (and that's how I use it most or the times). But writing clean Perl is
perfectly doable, though I usually prefer static languages if maintainable
code is a priority (no Perl, no shell, not even Python).

------
amirkdv
Years ago I spent a lot of time (arguably too much time) writing (ba)sh
scripts and this document was a great resource! But my main lesson from that
period was that bash pitfalls are readability/maintenance nightmares,
confounded by the fact that most people don't know about most of the pitfalls.
To make matters worse, you can't know which pitfalls are or aren't known to
the original author / future maintainer of the script in front of you.

But I'm not sure to what extent the language warts are inevitable consequences
of what makes it so expressive and powerful. For example: unix pipes are
amazing and the bash syntax for them is quite expressive. I can reproduce the
same behavior in any of the more "sane" languages but I find that the
equivalent code is a less readable procedural soup of string munging, control
flow, and error control.

------
asicsp
See also:

* BashFAQ - [https://mywiki.wooledge.org/BashFAQ](https://mywiki.wooledge.org/BashFAQ)

* shellharden: safe ways to do things in bash - [https://github.com/anordal/shellharden/blob/master/how_to_do...](https://github.com/anordal/shellharden/blob/master/how_to_do_things_safely_in_bash.md)

* Security implications of forgetting to quote a variable in bash/POSIX shells - [https://unix.stackexchange.com/questions/171346/security-imp...](https://unix.stackexchange.com/questions/171346/security-implications-of-forgetting-to-quote-a-variable-in-bash-posix-shells)

~~~
CaptainZapp
Especially the first link is incredibly useful to me.

Thanks a lot!

------
rashkov
Is there a greater principle behind all of these pitfalls, a mental model that
I can learn and use to avoid these kinds of mistakes? Or is it sort of like
CSS where years and years of ad-hoc development make it an exercise in
memorizing edge-cases?

~~~
Spivak
Some off the top of my head.

* On Linux file and directory names can be _anything_ except NUL. Because of how much software already assumes they’re UTF-8 you can probably safely assume it too and not run into much trouble. But since bash basically just does some text substitutions and then calls eval you have to be defensive.

* Know how bash parses front and back. Almost all the pitfalls are unexpected consequences of the parser not behaving how you expect. [https://web.archive.org/web/20200124085320/http://mywiki.woo...](https://web.archive.org/web/20200124085320/http://mywiki.wooledge.org/BashParser)

~~~
doctoboggan
Do you know if the `--` trick mentioned in the article to stop option scanning
works with all bash programs? I thought parsing was up to the program itself.
Is it just a widely used convention?

~~~
JdeBP
Talking of mental models: These are not "bash programs". Most of the ones
where you need this mechanism have nothing to do with the Bourne Again shell,
and are just generally-usable utility programs, usable even by things which
aren't shells at all.

They are, generally, programs that happen to use library mechanisms such as
(but not limited to) getopt() or the popt library to parse their arguments.

------
mey
[https://web.archive.org/web/20200907092725/https://mywiki.wo...](https://web.archive.org/web/20200907092725/https://mywiki.wooledge.org/BashPitfalls)

Since the site is being hugged a little too much.

I also really like
[https://github.com/koalaman/shellcheck](https://github.com/koalaman/shellcheck)

~~~
INTPenis
Seeing this wiki hugged to death by HN makes me realize that it would be
perfect for migrating to some simple cloud hosting and using static markdown
in git.

All their maintainers are already technical enough to handle doing updates in
git. And with certain static doc generators like mkdocs you can have an Edit
link in each page that brings the user to a Gitlab repo with an editor and
Preview.

~~~
pathseeker
Throwing everything at another company so you can blame your outages on them
does not a better engineer make.

This site is just doing something silly dynamically for something that should
be static (e.g. reading from a database). A single modern computer can easily
handle the HN hug of death serving static files.

------
simias
Most of the pitfalls described in this doc are /bin/sh pitfalls, not bash per-
se.

On that note, let me rant: stop writing bash scripts people. If POSIX shell is
not good enough, it's a good sign that you should move to a more expressive
language with better error handling. Bash is a crappy compromise: it's not as
portable as POSIX but it's also a very crappy scripting language.

You might say I'm a pedant because "bash is everywhere nowadays anyway, your
/bin/sh might even be bash in disguise" but I'll be the one having the last
laugh the day you have to script something in some busybox or bash-less BSD
environment. Knowing how to write portable shell scripts is a skill I value a
lot personally, but that might be because I work a lot with embedded systems.

Also if you assume that your target system will always have bash, there's a
very good chance that it'll also have perl and/or python too. So no excuses.

~~~
scipute68
This is basic nonsense. Design your scripts and provision your environments.
_You_ can stop writing bash scripts. I find them eminently useful and I
control/provision the environments they run in. If you are not in control then
do what you are told. Most people would puke if I told them that a lot of csh
is alive and well in sci-prod environments but that is the way it are. You
_do_ tailor your environment bc that is the way *nix was designed. Controlling
people and peoples habits is faang land.

------
1vuio0pswjnm7
"13\. cat file | sed s/foo/bar/ > file

    
    
       You cannot ..."
    

With sed, in some cases, you can. One of the differences between Linux-
provided sed and BSD-provided sed is the -a option.

FreeBSD manpage:

"-a The files listed as parameters for the "w" functions are created (or
truncated) before any processing begins, by default. The -a option causes sed
to delay opening each file until a command containing the related "w" function
is applied to a line of input."

So let's say you have a file

    
    
       cat > file
       sandfoo
       ^D
     

BSD-provided sed

    
    
       cat file|sed -na 's/foo/bar/;H;$!d;g;w'file 
    

The replacement is made without any temp file.

Note this example will add a blank line at the top.

Why use sed? sed is part of BSD base system, the toolchain^1 and install
images thus it can easily be utilised early in boot process. I am not aware
that "sponge" is part of any base system.

1\. NetBSD's toolchain can be reliably cross-compiled on Linux.

~~~
koala_man
An easier way to do the same thing but for any tool is `cat file | sed
's/foo/bar/' | sponge file`

~~~
yjftsjthsd-h
Good old moreutils:)

------
bonoboTP
The biggest transformative learning experience for me in Unix command gluing
was to understand argument vectors. That a command execution is not just one
big string, but will at the point of calling the command, be transformed by
Bash to a vector of strings, to be passed on to the execve system call.

The point is, all those quotes etc. are syntax we use to tell the Bash I
terpreter what the strings passed on should be. Think in terms of the string
vector and not in terms of the one big string in front of your eyes that is a
line of Bash code. The latter is a recipe for memorizing magic incantations
and cargo cult sprinkling of quotes here and there.

The horrible thing with Bash is that while normal programming languages make
the difference crystal clear between the name of a function being called and
its mandatorily quoted string args neatly separated with commas, Bash will
come up with this split on the spot using arcane rules.

Letting a string automatically fall apart into multiple argemts on the spot
may have seemed convenient in the old days for forwarding invocations, but it
seems clear to me that this was a design mistake.

In the end if you want to have robust Bash code you have to keep track of
string splitting and will almost never rely on the default behavior of letting
Bash "helpfully" pre-digest and split things up for you.

Much of Bash expertise is then about how to fight vigorously against the
default behavior of silently chunking up your args. But fighting really strong
means using e. g. arrays with ugly syntax, and other magic looking constructs.
It shouldn't take so much alertness to just keep to sane behavior. The problem
is, the interpreter is actively hostile to your endeavor and will explicitly
try to screw with you.

------
bluedays
Oh neat. I found a problem with a script I wrote today because of this.

I wrote a script recently that takes several screenshots from the same folder,
merges them using Image Magick and outputs the merged image into a PDF. Found
out through the first tip that I shouldn't be using ls:
[https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_....](https://mywiki.wooledge.org/BashPitfalls#for_f_in_.24.28ls_.2A.mp3.29)

------
slavik81
I was writing a script recently and I was going to use `cp -- "$src" "$dst"`
but neither `man cp` nor `cp --help` mentioned `--` as an option. I'm not sure
I'm using info right, but it doesn't seem to be mentioned in `info cp`,
either.

Is that feature described somewhere else, or is it just undocumented?

~~~
JdeBP
The -- mechanism in general, not specific to the cp command albeit mostly
limited to programs that use getopt/popt for their argument parsing, is
described _all over the place_. I first encountered it in a book, written by
Eric Foxley, entitled _UNIX for Super-Users_ and published in 1985.

* It's perennially asked about on _Unix and Linux_ Stack Exchange, ranging from [https://unix.stackexchange.com/q/11376/5132](https://unix.stackexchange.com/q/11376/5132) almost a decade ago to [https://unix.stackexchange.com/q/570729/5132](https://unix.stackexchange.com/q/570729/5132) last year.

* It's in a whole bunch of duplicate questions there, including (but not limited to) [https://unix.stackexchange.com/questions/linked/1519?lq=1](https://unix.stackexchange.com/questions/linked/1519?lq=1) .

* It's in the _Single Unix Specification_ , and has been for a long time. ([https://pubs.opengroup.org/onlinepubs/7990989775/xbd/utilcon...](https://pubs.opengroup.org/onlinepubs/7990989775/xbd/utilconv.html#tag_009_002))

* It's in the getopt(3) manual page from the Linux man-pages project. It's in the getopt(1) manual page from the util-linux project.

* It's in getopt(1) and getopt(3) in the NetBSD Manual. The same goes for the OpenBSD and FreeBSD manuals.

* It's in Blum's and Bresnehan's 2015 _Linux Command Line and Shell Scripting Bible_ , on page 377.

* It's in Marcel Gagné's 2002 _Linux System Administration: A User 's Guide_, on page 48.

* It's in Kirk Waingrow's 1999 _UNIX Hints & Hacks_.

* It was Q&A number 32 in part 2 of the Usenet comp.unix.shell FAQ document. ([http://cfajohnson.com/shell/cus-faq-2.html#Q32](http://cfajohnson.com/shell/cus-faq-2.html#Q32))

Specific to the cp command, or rather to the _many_ cp commands that you could
end up using, as this _is not a "bash command"_:

* No, the OpenBSD, FreeBSD, and NetBSD cp(1) manuals do not mention it.

* Neither does the GNU CoreUtils cp(1) manual page, as you noted. It's instead in the GNU CoreUtils doco section on "common options". The actual CoreUtils cp(1) manual doesn't mention that there _even are_ "common options". ([https://manpages.debian.org/sid/coreutils/cp.1.en.html](https://manpages.debian.org/sid/coreutils/cp.1.en.html)) That's only mentioned, as an aside, in the info page. ([https://gnu.org/software/coreutils/manual/html_node/cp-invoc...](https://gnu.org/software/coreutils/manual/html_node/cp-invocation.html#cp-invocation))

* For the cp command that is part of Rob Landley's toybox, this is covered in the introduction to the toybox(1) manual page, and there isn't really a distinct cp(1) manual page. ([http://landley.net/toybox/help.html](http://landley.net/toybox/help.html))

* No, BusyBox doesn't similarly document this for _its_ cp command, or anywhere on busybox(1). ([https://busybox.net/downloads/BusyBox.html](https://busybox.net/downloads/BusyBox.html))

* It's tangentially in several GNU CoreUtils FAQ Q&As, inasmuch as CoreUtils cp is the same as CoreUtils rm in this regard. ([https://gnu.org/software/coreutils/faq/coreutils-faq.html#Ho...](https://gnu.org/software/coreutils/faq/coreutils-faq.html#How-do-I-remove-files-that-start-with-a-dash_003f))

* It's tangentially in the Linux Documentation Project's _Advanced Bash-Scripting Guide_ , again because cp is the same as rm here. ([https://tldp.org/LDP/abs/html/special-chars.html#DOUBLEDASHR...](https://tldp.org/LDP/abs/html/special-chars.html#DOUBLEDASHREF))

* For an operating system where this is actually in the cp(1) manual _itself_ , look to Solaris, and its successor Illumos, where you'll find it in the NOTES section. ([https://illumos.org/man/1/cp](https://illumos.org/man/1/cp))

* Another operating system where this is actually in the cp(1) manual _itself_ is AIX, actually in the FLAGS table alongside all of the other flags. ([https://ibm.com/support/knowledgecenter/ssw_aix_72/c_command...](https://ibm.com/support/knowledgecenter/ssw_aix_72/c_commands/cp.html))

~~~
slavik81
That is a far more comprehensive answer than I was expecting!

You're absolutely right about my system. There is a link to the coreutils
common options in the cp info page. I missed that when I was searching through
it.

------
ryangittins
Looks like HN has hugged this one to death.

[https://web.archive.org/web/20200907174629/https://mywiki.wo...](https://web.archive.org/web/20200907174629/https://mywiki.wooledge.org/BashPitfalls)

------
chousuke
One new pitfall I learned yesterday is that sourcing a file searches $PATH. I
ran a tool that did ". config" on OpenBSD and it tried to source config(8)
instead of its configuration in the local directory. Oops.

~~~
JdeBP
On the subject of OpenBSD, sourcing configuration files, and security: Enjoy
[https://unix.stackexchange.com/a/433245/5132](https://unix.stackexchange.com/a/433245/5132)
.

------
rurban
I have almost no bash scripts.

Whenever sh is not good enough and I have to switch to bash, I'd rather
rewrite it in proper Perl. Bash really looks like a hack, and not all
environments have bash. Everybody has a conforming sh.

------
LockAndLol
I was half-expecting a page with the bash grammar. It's super easy to make
mistakes in that language. I'm glad there are things like xonsh around trying
to act as sane alternatives.

------
znpy
It's both interesting to read and meh.

It's meh because like many things, you actually have to read the fine manual
if you want stuff to actually work.

It's interesting because it's always nice to have a refresher and/or reminder
of various bash things.

That being said... I've said this many times, I'll say it again: bash scripts
are software, and thus must not be exempted from various programming best
practices. The first thing that comes to my mind is input validation and error
checking.

Bash is not magical. It's one of those things that have to be mastered through
practice... Just like many other programming languages.

------
inoffensivename
It would be easier to list all the constructs in bash that are well-behaved
and intuitive.

------
zerofrancisco
Excellent caveats and suggestions to do good bash. Thank you! :)

------
phibz
I wonder if there's a linter to check for these issues.

~~~
boogies
Shellcheck?
[https://github.com/koalaman/shellcheck](https://github.com/koalaman/shellcheck)

------
baby
To me this screams: don't use bash in production.

~~~
jakeogh
Yea. Or test the heck out of it. Once it works, it tends to work well.

It took me ~8 years (so far, I'm sure there is a bug somewhere) and
considerable #bash advice to make a simple script that locks commands+args
[https://github.com/jakeogh/commandlock](https://github.com/jakeogh/commandlock)

I also wrote a MDA wrapper for gnupg, and again, it took years to feel good
about having my mail filter through it
[https://github.com/jakeogh/gpgmda](https://github.com/jakeogh/gpgmda)

Now that py has pathlib, if it's more than a few lines, py with @click is just
way better. All that is really missing is cleaner copy/move abstractions.

There are also some features that are missing from all shells... like the
ability of a script to know if expansion happened before it got $@... that
internal state just isn't exposed.

------
hi41
What is the currently accepted best practice to write scripts? Is it to use
Perl or Python?

------
kennywinker
Bash pitfall #1: deciding to use bash

~~~
jjgreen
So true, bash is a fine interactive shell, but for scripts the Bourne shell is
almost as capable, portable as in POSIX and did not suffer shellshock.

