Better than set -e is trapping ERR. You can set up a trap that prints out the command that failed, and the line it's on, rather than just dying quietly like set -e does. It's much easier to debug that than trying to work ahead from the last command with output. For bonus points, when you have a bunch of scripts together, put the trap code in its own file and source it in all the other scripts (rather than duplicating code).
The set -e option instructs bash to immediately exit if
any command has a non-zero exit status. You wouldn't
want to set this for your command-line shell,
...but, it's a great addition to your buddy's `.bashrc`. For maximum effectiveness, be physically present, perhaps with a camera. :)
I don't think that setting IFS this way is a good idea. Of your variables do happen to contain tabs and newlines, you still get unwanted word expansion. Much better is to just use double quoted expansion primitives that always expand one element to each word: "${foo[@]}"
That's the common answer, and while I practice that myself, I have to disagree with it as guidance. Maybe you and I are fastidious enough to always remember to quote all our variables, but many are not - I know I was writing shell scripts for a couple of years before I realized its importance, and it's very common that even experienced engineers don't know or care to do it. If someone writing or editing the code forgets to quote the variable that allows subtle bugs to sneak in.
It's unfortunate the semantics of bash don't have variable references behave like they are quoted by default. I really wish it did.
If you can remember to use $@ rather than $*, you should be able to remember to use "$@". The only reason $@ exists is that it behaves differently when enclosed in double quotes.
While setting IFS=$"\t\n" may make problems less likely when strings contain spaces, it's still not correct. File names (and other strings) can contain tabs and newlines, too. That's relatively rarer, but the quoting approach always works.
#!/bin/bash
items=(
'a'
'b c'
"d\te"
"f\ng"
)
echo "Unquoted:"
for item in ${items[@]}; do
echo -e ". $item"
done
echo "Quoted:"
for item in "${items[@]}"; do
echo -e ". $item"
done
set -euo pipefail
IFS=$'\n\t'
echo "Unquoted strict mode:"
for item in ${items[@]}; do
echo -e ". $item"
done
... I get this output:
Unquoted:
. a
. b
. c
. d e
. f
g
Quoted:
. a
. b c
. d e
. f
g
Unquoted strict mode:
. a
. b c
. d e
. f
g
Note the output for "Quoted" and "Unquoted strict mode" are identical.
(GNU bash, version 4.2.37(1)-release (x86_64-pc-linux-gnu))
Can you give an example of an exploit in bash? The link you give is excellent information, but doesn't have any shell script examples.
My first thought: It seems like only $@, $* and variables read from the environment could be possible attack vectors. But since shell scripts can't be setuid, it's hard for me to immediately imagine how unquoting could enable a new exploit.
EDIT: the most dangerous example I can think of right now is a script like this, where a webapp invokes the script and passes a GET param as an argument:
with less than three arguments passed and shift_verbose on you will get an
error message the moment you shift and with "set -e" in addition, execution
will be aborted.
Also, "set -o noclobber" might be useful. If you try to
redirect to an existing file with ">", it will fail. If you explicitly want to overwrite the
file without triggering the error, use ">|".
When using `set -eu`, how do you check the number of arguments?:
if [ -z "$1" ]; then
usage();
exit 1;
fi;
If I'm using `set -eu`, that dies on the `$1` with a nasty error message, rather than printing my usage message. I've resorted to moving `set -eu` to after these kind of checks, but that makes me uneasy.
In case it matters, michaelmior and Splognosticus's solutions are both POSIX shell compliant, so they should work anywhere.
(eg, busybox, dash, tcsh, zsh)
Good info for writing bash scripts that are large enough to need debugging, but why would one do that in the first place? Python or Perl would be better choices at that point.
When you are fundamentally running shell commands, using python/perl really don't make a lot of sense. Use the best tool for the job. Note that for my dayjob I write python almost fulltime, but if I'm almost exclusively running shell commands, I'll write a shell script. Just because you can do something one way doesn't necessarily mean I should.
See I'm of the opinion that if you need arrays and associative arrays, bash is the wrong tool for the job. If you have a recent bash it has both of those, just seems wrong in such a clunky language with awful scoping.
I write a lot of bash scripts, but lately I've been doing most of my shell interop in Ruby instead. Backticks are pretty good and I can munge things in easier ways than bash or Python. And with `ruby -n`, the script is invoked line-by-line--perfect for processing piped content.
Use the best tool for the job. If you don't know bash/bourne shell well, use ruby/perl/python/etc. I started out as a sysadmin years ago and know bourne shell/bash very very well. It is all about what works best for the problem.
Sure. And don't get me wrong, I write a lot of bash scripts. =) I think any logic beyond string replacement is probably edging out of where it's a good idea, if only because other people then have to read my stuff later, but it's totally fine for that. I'm saying more that I think Ruby (or Perl) make more sense than Python given the tools it provides.
Bash is not just Bash but also sed, awk and all the other goodies. You can grow Bash scripts from the command line. You don’t have to worry about things, everything is a string is a number is a stream. One could go so far as to argue that shell pipes behave quite distinctly functional-programming-ly(?).
Plus it’s some sort of middle ground between “BDFL tells you to put a space there“ and “one day, archaeologists will find a script written in Perl, awk and csh and it will become the new millenium’s Rosetta stone“.
It's happened more than once that I've started something in bash, thinking it's a small enough job, only to see it growing in time - a lot. Sure, in theory you could rewrite it in Python, but you don't always have that luxury in practice.
Bottom line is - keep bash scripts small, but be prepared to deal with the cases when they grow like Jack's magic beanstalk.
because then you can capture newlines and tabs. Bash will automatically convert it to separate parameters, even though there is only one quoted variable.
I wonder why not to set IFS='' then. I haven't used this myself in production, but quick testing seems to do what I expected: make $foo behave like "$foo".
