
Writing Safe Shell Scripts (2019) - rrampage
https://sipb.mit.edu/doc/safe-shell/
======
mekster
Make sure to have ShellCheck either integrated in your editor or run it before
executing.

It really tells many of the rules you're supposed to abide by and helps you
write cleaner shell script.

[https://github.com/koalaman/shellcheck](https://github.com/koalaman/shellcheck)

~~~
urda
Quick reminder that ShellCheck is licensed under the strict GNU GPL 3.0
license. For many professionals your employer will often block / avoid GPL
code, tools, and libraries.

~~~
ahnick
Sorry, but what employers block the use of GPL tools? Most employers that
write proprietary software don't want GPL code in their code, but using a tool
like shellcheck to perform static analysis on your code doesn't make that
software that was analyzed now subject to the GPL, so why would they block it?

~~~
bradknowles
Apple is strongly anti-GPL. And they’re pretty open about that.

Most companies that are anti-GPL aren’t so open about it.

~~~
ahnick
I get that Apple doesn't want GPL code included in their software, but their
engineers can still use GPL tools without fear of opening up their proprietary
code. I know of plenty of companies where you can not _INCLUDE_ GPL code in
the software the company makes, but I don't know of any company that says you
can not _USE_ any GPL software to do your job. Those are two different things.

~~~
bradknowles
For the six month contract I did at Apple, they were violently opposed to
using any GPL software in any way, shape, or form.

Just because you can do something, doesn't necessarily mean you should. That
rule applies to the implementation of corporate policies regarding licensing,
too.

------
vaingloriole
This is the kind of ignorance (just use python for anything it's better + and
discover subprocess(): it's great) that burns me up. Not so many years ago
actively insisting that python/tcl/perl were to be used instead of a 50 line
shell script would have gotten you kicked out of a serious discussion. You
used the power and reflection of these languages when you needed something
that didn't integrate cleanly with the rest of the unix toolkit and/or
required sophisticated data structures, controls and abstraction. These days I
won't even write python or tcl for trivial projects (including http API work
just use curl/jq and bash/(g)awk). The number of shell one liners that just
get things done in 30 seconds (rather than loading 5 libraries in python to
write the same code) are innumerable. This is google thought policing the
historically ignorant yet again. They are terribly opposed to the unix toolkit
and C pragma that prevailed for 20 years...because they believe that they know
the best way to make the world safe and productive. Self serving, smug
rejections that serve the bottom line and coincidentally gain mind share for
their products and approaches.

~~~
0xff00ffee
Hear hear! It tears at my heart that the next generation of admins think
Python is the answer to everything. It's a systemic rot of the core of *nix
understanding.

I'll probably get flamed for this, but Python is an absolute fucking
trainwreck of a platform. Even at its core the 2.x vs 3.x is going to break
all kinds of stuff in Jan 1, 2020. Just look at the package system: cross-
system portability is a mess (don't believe me? run on Arm for a week), and
unfortunately because there IS a package system people think they HAVE to use
it, which means: BLOAT BLOAT BLOAT your BOAT, gently down the stream!

It's kinda like what Dewalt did to miter saws. We've gone from something that
could be kept perfectly square and did a great job and took up a little bit of
space, to 30+kg behemoths with sliding rails and double bevels, tons of blade
travel and deflection, and are a challenge to keep true and take up 3x the
space and weigh a ton.

Keep your mission-critical tools simple. Period.

------
lalaland1125
I really wonder whether there is really any point in writing shell scripts
anymore. Practically every Unix/Linux box in existence has at least some
version of Python 2 that can be used as a complete and total replacement. I
can't think of a single situation where I would need a shell script and a
Python script wouldn't be much cleaner, simpler, and more maintainable.

~~~
heavyset_go
I've yet to find a programming language that makes I/O redirection, piping,
and process substitution[1] as easy as Bash does. Process substitution is
where the shell really shines, in my opinion.

Bash, and Bash-like shells, are literally everywhere. I have to be wary about
what Python 3 features I use, and if there will even be a Python interpreter
available. My OpenWRT router has a Bash shell, but I don't care to install and
maintain other interpreters or runtimes on an embedded platform.

Even job management and parallelism are easier in Bash and GNU Parallel.

[1] [https://www.tldp.org/LDP/abs/html/process-
sub.html](https://www.tldp.org/LDP/abs/html/process-sub.html)

~~~
Pxtl
PowerShell, mostly because PowerShell was designed as a mashup of Bash and
C#.... And it's kind of a trainwreck in a lot of ways.

It really feels like piping and easy process invocation and compile-time
directory awareness wouldn't be massively onerous to add to an existing full-
featured programming language so you wouldn't have to sacrifice a good type
system and powerful syntax when you want to do scripty things.

~~~
chrisfinazzo
As I understand it, PowerShell came about because porting the standard
Linux/Unix utilities to Windows didn't work out.

PowerShell meets both requirements as it's tuned for the Windows environment,
but can still play nicely with the rest of the world.

Of course, now that WSL is a thing, it's a real question why someone would
want to use PS unless they had no other options.

[https://www.heavybit.com/library/podcasts/to-be-
continuous/e...](https://www.heavybit.com/library/podcasts/to-be-
continuous/ep-37-the-man-behind-windows-powershell/)

~~~
sjy
PowerShell has real data structures (arrays and hash tables), built-in
functional programming tools (Select-Object, ForEach-Object, Where-Object),
GUI cmdlets (Out-GridView) and direct access to .NET libraries. It’s a viable
tool for many complex tasks where on Unix, you’d need to reach for Python
rather than a shell.

------
smartmic
Shell scripts have their well-deserved place in Unix systems. A general
recommendation to use Python or other high-level scripting languages without a
discussion about the reasons why and when to use shell may lead to wrong
conclusions.

~~~
savolai
I'd love to read a good blog post on when to use shell scripts vs Python or
other programming languages, which seem far more accessible to me.

~~~
peterwwillis
If all you need to do is chain together external programs, set environment
variables, redirect filehandles, iterate over files, etc with the tiniest bit
of logic, and don't want to have to set up an execution environment or
download something to do it, and want ultimate Unixy portability, and you want
virtually anyone to be able to read and maybe modify it, you want shell.

If you need to do a very specific task that involves interpreting/modifying
data structures/formats, if you have non-trivial logic, if you need to use a
module, if you need to access an operating system primitive which Unix shells
don't really expose, if there's no Unix tool that does what you want, or you
just need more control/certainty/reliability, you want Python or something
else.

~~~
savolai
The funny thing though is that the "virtually anyone" part seems to me more
likely to happen with python - or with any mainstream programming language -
than with what seem to me like deeply arcane shell incantations.

------
tjoff
I'm not sure I'm comfortable with python for a typical shell script. But then
again, what alternatives exist?

I've been thinking that we are kind of stuck with it. Much like javascript.

So, we could go the route of typescript and have a translation layer and
outputting shell scripts. I actually think that could work quite well from a
technical standpoint.

A big part of shellscripts though is having the source available. So maybe
even have the translated, original code, and the generated shell output in the
same file.

The source code on top (out of view from a shell interpreter) and the
generated code below. This would allow it to be readable by anyone and
executable by anyone, but modifying it would require the transpiler (which by
itself would probably be pretty lightweight as well - certainly compared to
python).

Certainly not perfect, but preferable to rediscovering the nuances and gotchas
of shell-scripts every other day. Or maybe such a solution would only prolong
the suffering and we should start all over.

~~~
Bnshsysjab
What’s wrong with python?

~~~
tjoff
Feels like the wrong tool for the job with the whole virtual environment that
has to be brought up.

Performance for shell scripts isn't super critical in my eyes but the startup
time of python feels too much for something you might want to run in an inner-
loop from find or something. Not sure if python IO performance is suitable for
shell scripts either.

Also isn't nearly as universal as shell-scripts nor something you necessarily
have/want on small systems.

But I will admit that I'm channeling prejudices and don't feel I have enough
of a complete picture to say anything definite about it. Others in this thread
touch upon it though.

~~~
Bnshsysjab
You don’t _need_ a virtual env.

I probably wouldn’t wrap GNU find and python, I’d just walk the directory tree
in python and use regex.

IO is suitable for everyday tasks and shouldn’t be an issue in Python anyway,
if it is either you’re doing it wrong or your shell script should be written
in C and not touch interpreted/bytecode compiled languages at all.

~~~
tjoff
Me neither, but a big point of shell scripts is that they are the same
commands you'd run on the shell itself.

And there are lots of really optimized tools that are as quick as can be,
throwing them out is a hefty price to pay.

------
arendtio
Some time ago I started to doubt if using 'set -e' is a good idea.

I mean, if it would work as you expect it to, it certainly is a good idea to
exit a script as soon as something fails. But sadly not all implementations
behave similarly [1] and if you call a function from within a condition, 'set
-e' gets deactivated/doesn't work.

For illustration, take a look at the following example:

    
    
      #!/bin/bash
      
      foo() {
        set -e
        false
        echo "Sucks"
      }
      
      printf 'Normal: %s\n' "$(foo)"
      printf 'Condition: %s\n'  "$(foo || true)"
    

Output:

    
    
      Normal: 
      Condition: Sucks
    

Probably not what you would have expected. Not using 'set -e' is no good
solution either, so, for the time being, I still use it, but I am still
looking for a better solution.

[1] [https://www.in-ulm.de/~mascheck/various/set-e/](https://www.in-
ulm.de/~mascheck/various/set-e/)

~~~
shakna
If I have something I know can fail, and I want it to be able to, then calling
set +e before it, and recalling set -e afterwards is the way I handle it.

But, in this case, because you're running it in a subcommand, you need to add
pipefail.

    
    
        set -euo pipefail
    

This way of making sure nothing misbehaves also locks out unset vars, which
can be surprising, but also helpful.

~~~
arendtio
As far as I can tell, adding pipefail or nounset doesn't change the behavior
in this case.

In my opinion, the problem is that calling a function from a test expression,
evaluates it in a different manner than calling it from elsewhere. That is so
absurd that I wonder how it wasn't changed over the years.

------
chousuke
Can mktemp fail? I recently wrote a script that ensures that the directory
created by mktemp actually exists and starts with /tmp so that when the script
wipes out its temporary data at the end, it will not ever run something like
"rm -rf /". Using fully-qualified paths for everything is also (not) fun.

In another script I was leery about running rm -f -- /path/foo* so I instead
used

    
    
      find /path -mindepth 1 -maxdepth 1 -name "foo*" -delete
    

because then the glob can't affect find's behaviour in weird ways.

Most likely set -e is enough protection against mktemp failing and the rm -f
glob _probably_ would've been just fine, but shell scripting has enough
footguns that sometimes I can't help but go overboard with paranoia.

~~~
TheDong
Yes, mktemp can fail.

For example, "TMPDIR=/dev/null mktemp -d" will reliably fail.

You shouldn't be validating it starts with "/tmp" though, because on quite a
few systems, people set TMPDIR=/var/tmp.

You absolutely should check if mktemp exited nonzero, but if it exited 0, the
directory should be safe to use. If you want to be really paranoid, you can
check if the output is an empty string too after checking the exit code.

I think you're shying a little too far away from bash's globbing. Yeah, bash
has footguns, but using clever workarounds can also be a recipe for creating
confusing and error prone code in its own way.

~~~
emj
Please tell me it's enough to check the exit code! That's all I ever do and I
have hundreds of mktemp scripts out there.

In the end I think shelling is mostly about controlling your inputs.

~~~
clarry
Not just inputs, unfortunately. TOCTOUs (and races in general), for instance,
are very common problem in shell scripts.

~~~
arpa
Races and TOCTOUs are generally common problems in most concurrent systems
that are not considering atomicity by design (e.g. rdbms). It's just that
shell can very easily become concurrent...

------
derefr
I’m constantly surprised that we got a safe language that compiles to
Javascript (TypeScript) but never got a safe language that compiles to shell
script. Why can’t I write in (a subset of) some other scripting language
$lang, but with restricted-to-pure-/bin/sh semantics, and then cross-compile
it to actual portable shell script for distribution?

Hopefully not by generating a megabyte of polyfill runtime code; more just by
the compiler refusing to compile code in $lang unless it has a direct
equivalent in /bin/sh. Any $lang source file the compiler accepted, would just
look like “a shell script translated into $lang” already. But the compiler
would inject $lang’s safe semantics (exceptions, type-checking, etc.) during
the compilation, so you’d at least get that benefit.

Alternately, I’d _also_ be satisfied with “emscripten for shell”: a compiler
that generated shell scripts containing a small WASM-like emulator and an
embedded bytecode stream to feed it. As long as it was lightweight enough. (A
large point of these environments’ use of /bin/sh is that they’re small and
embedded and can’t load too much into memory at once.)

My only guess for why this hasn’t happened, is that the people who write
things like shell scripts that execute in initramfs, or shell scripts that are
intended to install stuff even on lesser-known UNIXes, are all old-hand ops
people who know shell-scripting cold, and certainly _aren’t_ developers with
any experience in compiler theory.

~~~
cdaringe
Because doing so would be hard for shell scripts to express. real functions,
try/catch, and other common language features are not often part of shell
languages. As such, the generated code may not support many features you typed
in <other lang> and/or the generated code would bundle a runtime with it to
emulate what we'd otherwise consider pragmatic scripting feature support.

~~~
derefr
I mean, like I said, I don’t want to use $lang features that don’t exist in
shell script. E.g., I want “exceptions” in the sense that adding a string-
typed variable to an integer-typed variable will blow up in a descriptive way
(or better, not compile); but I don’t want real exceptions in the sense of
being able to catch them. I just want everything to translate to the shell
script aborting in verbose and helpful ways before doing something stupid with
invalid inputs; or not compiling at all if there’s a code-path that can’t
possibly be valid.

Also, I wouldn’t mind if this $lang that compiles to shell script is its very
own language I’d have to learn, just like TypeScript is its very own language
you have to learn. As long as I don’t have to manually write five layers of
guards using impenetrable [ “y${x:-1}” -eq “f” ] style code, and then
duplicate the hierarchy of cleanup behaviours after each failed guard, I’d be
happy.

------
tyingq
Checking $PATH or explicitly calling outside resources is probably also
advisible.

In general, this article seems very light. No mention of LD_LIBRARY_PATH, for
example. I imagine there's probably a better guide to writing secure scripts
somewhere else.

~~~
lalaland1125
These sorts of security measures are only really necessary when you have a
setuid or setgid shell script. (However, a much better suggestion would be to
simply not write setuid or setgid shell scripts because they are a security
nightmare.)

------
loevborg
For more concrete recommendations in a similar spirit, check out
[https://github.com/pesterhazy/blissful-
bash](https://github.com/pesterhazy/blissful-bash)

------
moreati
Instead of

    
    
        set -euf -o pipefail
    

please consider

    
    
        set -o errexit
        set -o nounset
        set -o noglob
        set -o pipefail
    

which is easier to lookup/search for if the reader is less familiar with shell
scripting, and makes for cleaner diffs when a flag is removed/added.

Caveat: some of the longer forms might be Bashisms (e.g. not present in ksh,
dash, etc.)

~~~
TheDong
"which is easier to lookup/search for"

From my experiment:

Googling "set euf" (without quotes) -- 838,000 results, first 6 results are
all helpful and explain it in detail.

Googling "set o nounset" (without quotes) -- 38,700 results, entire first page
seems helpful.

I dunno, looks like they're identically easy to learn the meaning of.

~~~
moreati
"easier to understand/remember the meaning of" would have been better
phrasing. I can remember/infer what `set -o noglob` does, but not `set -f`.

------
ilyash
Hi. Author of Next Generation Shell here.

Here is my take on bash vs Python and friends. I am frustrated by both.

bash - domain specific language and huge library (CLI utilities) and lets you
get the job done but not a language I would like to use (syntax, error
handling, very limited structured data support).

Python - okayish as a language but doing domain specific things (working with
files and processes) is so much more verbose than bash.

My solution in NGS - have a _high level, modern_ language with the typical
goodies like ... exceptions (wow, completely new concept!) and the language is
still domain specific. Working with processes for example has it's own syntax
and is much more concise and straightforward (stole bits from bash).

Right now the project has the language, which is useful enough to write
scripts (which we do at work). Regarding contribution to the project, my idea
is that as much as possible should be in NGS language (as opposed to the lower
level C). The UI will be implemented completely in NGS, allowing contributing
to the project using the _same language_ you are writing your scripts in.

[https://github.com/ngs-lang/ngs](https://github.com/ngs-lang/ngs)

As a side note, while the UI is not there yet, I do plans for it which include
interaction with objects on the screen as opposed to current "here is your
dump of text" situation... and in general be more considerate of the user.

[https://github.com/ngs-lang/ngs/wiki/UI-Design](https://github.com/ngs-
lang/ngs/wiki/UI-Design)

------
jakeogh
I find exceptionally difficult to write anything but trivial shell scripts
without bugs. This one took years, and I suspect #bash could still find a bug:
[https://github.com/jakeogh/commandlock](https://github.com/jakeogh/commandlock)

On the other hand, this is amazing: [https://github.com/speed47/spectre-
meltdown-checker](https://github.com/speed47/spectre-meltdown-checker)

------
joana035
Go write a shell script just like any other piece of code. Have fun!

------
cr4zy
Here's my boilerplate bash header with comments describing each option, which
I find practically helpful for starting new scripts

[https://gist.github.com/crizCraig/f42bc250754bed764ada5f95d1...](https://gist.github.com/crizCraig/f42bc250754bed764ada5f95d101dbea/)

raw:
[https://gist.githubusercontent.com/crizCraig/f42bc250754bed7...](https://gist.githubusercontent.com/crizCraig/f42bc250754bed764ada5f95d101dbea/raw/e3814fdb85c42afcf5766abf039f51eed67d8068/bash_boilerplate.sh)

------
arglebargle123
I've done a bit of porting bash scripts to posix recently and I've found the
following boilerplate pretty useful:

    
    
        case "$(readlink /proc/$$/exe)" in */bash) set -euo pipefail ;; *) set -eu ;; esac
    

That takes care of setting -eu for shells that don't support -o pipefail and
pipefail for bash.

------
gugagore
I personally think "don't" is right. For those who are interested in trying to
use Python instead, I found this to be a helpful resource:
[https://github.com/ninjaaron/replacing-bash-scripting-
with-p...](https://github.com/ninjaaron/replacing-bash-scripting-with-python)

~~~
stebann
I disagree, Python scripting will tie you to Python development uncertainties,
while shell works with most operational environments.

~~~
montyhallpy
That's a good point. Shell would not announce an EOL date for a specific
version. Scripts that used to work 25 years ago, still work just the same.
Python scripts that used to work 10 years ago, now need maintenance/porting,
no matter how trivial, to make them work with Python 3.

------
muth02446
More useful tips in the same spirit

[http://robertmuth.blogspot.com/2012/08/better-bash-
scripting...](http://robertmuth.blogspot.com/2012/08/better-bash-scripting-
in-15-minutes.html)

------
liopleurodon
related: [http://redsymbol.net/articles/unofficial-bash-strict-
mode](http://redsymbol.net/articles/unofficial-bash-strict-mode)

------
vbernat
Since it's bash-specific, I think people should use zsh instead. No need to
quote variables as by default, splitting is not done. And you get access to
arrays and associative arrays.

~~~
codys
Bash has arrays.

    
    
        x=(1 2)
        echo "${x[0]}"
    

And associative arrays

    
    
        declare -A x
        x[hi]=20
        x[bye]=30
        echo "${x[hi]}"
        for i in "${!x[@]}"; do
            echo "k: $i, v: ${x[$i]}"
        done

------
tomohawk
On set -e:

This is like adding a setting to your Java or Python program to immediately
exit if any method call throws any kind of exception, with no possibility of
catching and handling that exception.

This does not seem to be a smart thing to do as compared to checking return
codes, etc.

Every script I've seen with set -e has been buggier than without it.

The main problem with scripts is that they often do not go through a normal
software review and testing process.

------
DonHopkins
The first piece of advice, "Don't", is the best. "shellcheck myscript.sh"
should return a fatal error if the shell script file you're checking exists,
and "shellcheck -f myscript.sh" should fix your shell script by removing it.

------
cdaringe
Amen.

deno, node, python, julia--anything but shell scripting.

------
alexis_fr
At the beginning of each file:

    
    
        #!/bin/bash
        set -euf -o pipefail
        cd $(dirname $0)
    

Then add "" everywhere ;) And they say, write Python instead, except I’m
dubious because python programs can have a lot of dependencies which can be
tricky to install.

~~~
TheDong
Your recommendations are a little off in the following ways:

You should use /usr/bin/env bash for the shebang. Some distros, such as nixos,
don't have /bin/bash.

You need more quoting in the dirname line. cd "$(dirname "$0")" is what you
need. The outer quotes are in case the directory has a space or special
character in it, the inner ones are in case the script does.

That being said, you may also want $BASH_SOURCE rather than $0, but the
reasons behind that are complicated. You may also want to support cases where
the script is a symlink depending on what you're doing.

~~~
arglebargle123
Using 'env bash' to find the local path is definitely the way to go. It also
supports that occasional case where the user has installed modern bash
somewhere in their local path (under their home directory) and finds that
rather than the system copy.

    
    
        #!/usr/bin/env bash
    

If you really want a predictable environment I usually go with something like
this after the shebang, it supports bash and not-bash shells:

    
    
        case "$(readlink /proc/$$/exe)" in */bash) set -euo pipefail ;; *) set -eu ;; esac
        PATH=/usr/sbin:/usr/bin:/sbin:/bin
        \unalias -a
    

I don't usually cd anywhere unless I need to.

------
seorphates
Not to be that guy but this is mostly garbage. I suggest simply paying
attention and testing. Switches can be a useful part of the tool but they can
break things. It doesn't make things better rather it alters behaviors of the
shell in sometimes unintended ways. Try some loops with some -eo. If you have
a pipe that fails your script you're just ripping the heart and potential out
of your script. I think you might just be looking for one-liners. It can be
good to catch your failures and try other things vs "woop, damn, nope." Maybe
you need things to fail first.

"Quote liberally" \- also ripe. Mind your aPostrophes and Quotes because they
do things.

I like writing for me and what I need done, not what or how others might have
me write or how others might think it should be done.

The shell is a base orchestration and interface tool for your os. If you think
it's just that bad then just stick to your language of choice but please
refrain from suggesting lazy habits make a better shell, they don't.

Fly your own kite. It's much more fun.

~~~
seorphates
Ok, "garbage" may have been harsh but how about a fun example? Try this with
and without -e

    
    
      #!/usr/bin/env bash
    
      bell=`tput bel`
      tock='Blastoff!'
    
      do_ring()
      {
        if [ "$1" ]; then # true
          echo -n $bell; sleep 0.1
          echo -n $bell; sleep 0.1
          echo -n $bell; sleep 0.1
          echo $tock
        else
          echo -n $bell; sleep 0.1
        fi
      }
    
      i=3
      while [ "$i" -ge "0" ];
      do
        if [ $i = 0 ]; then
          sleep 0.5
          echo ok
        else
          echo $i $bell
          sleep 0.5
        fi
        i=`expr $i - 1`
      done
    
      do_ring $1 # do something different if arg
      sleep 1
      echo neat.
      
    

Semantic qualities aside there is something here that breaks with -e and if
you're used to using -e by default you might be baffled and think shell sucks,
for example.

~~~
clarry
That's why nobody uses expr for arithmetic. But yes, it is annoying that -e
breaks some habits that work fine without it.

~~~
seorphates
I can appreciate the quip, especially given my curmundgeonly entry. But the
point remains. The shell needs to be able to call any command on the system
reliably. Even aged old counting methods. At no point in that loop would the
expr subprocess return other than zero. Try to break out of that loop with
that bash switch.

One should not rely on these magics, the ones that alter runtime behaviors,
without truly understanding what you're after and what you're writing. Or
mostly understand. They're fine and good if you have narrow routines with a
very strict focus - "I'm in, I'm out, bang."

The best switches available to the shell without a bunch of feature and
package and library this or that bloat are exactly

    
    
      -x
      -n
    

That's all that you need to ensure that your scripting is what you need it to
be. Introduce the magic after you've written the thing. When it's ready and it
passes your tests then test it again.

Every single utility and package on that system is at your fingertips. Demand
accuracy. I do.

